[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 743098627 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.002409] x2apic enabled [ 0.003015] Switched APIC routing to physical x2apic. [ 0.004028] kvm-guest: setup PV IPIs [ 0.007885] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009023] pid_max: default: 32768 minimum: 301 [ 0.010256] LSM: Security Framework initializing [ 0.011082] Yama: becoming mindful. [ 0.012059] SELinux: Initializing. [ 0.013133] *** VALIDATE selinux *** [ 0.022792] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027519] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028252] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029209] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030176] *** VALIDATE tmpfs *** [ 0.032540] *** VALIDATE proc *** [ 0.033342] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036301] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037199] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040037] Spectre V2 : User space: Vulnerable [ 0.042006] Speculative Store Bypass: Vulnerable [ 0.046166] debug: unmapping init [mem 0xffffffff92259000-0xffffffff92260fff] [ 0.048295] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049996] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051041] ... version: 2 [ 0.052022] ... bit width: 48 [ 0.053020] ... generic registers: 4 [ 0.054030] ... value mask: 0000ffffffffffff [ 0.055026] ... max period: 00007fffffffffff [ 0.056030] ... fixed-purpose events: 3 [ 0.057022] ... event mask: 000000070000000f [ 0.058460] rcu: Hierarchical SRCU implementation. [ 0.061135] smp: Bringing up secondary CPUs ... [ 0.062796] x86: Booting SMP configuration: [ 0.063041] .... node #0, CPUs: #1 #2 #3 [ 0.072021] smp: Brought up 1 node, 4 CPUs [ 0.074019] smpboot: Max logical packages: 1 [ 0.075033] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.112070] node 0 deferred pages initialised in 35ms [ 0.117311] devtmpfs: initialized [ 0.118372] x86/mm: Memory block size: 128MB [ 0.121420] gcov: version magic: 0x41383552 [ 0.123561] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.124268] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.125337] pinctrl core: initialized pinctrl subsystem [ 0.126343] [ 0.126930] ************************************************************* [ 0.127022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.128021] ** ** [ 0.129023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.130022] ** ** [ 0.131022] ** This means that this kernel is built to expose internal ** [ 0.132026] ** IOMMU data structures, which may compromise security on ** [ 0.133024] ** your system. ** [ 0.134023] ** ** [ 0.135022] ** If you see this message and you are not debugging the ** [ 0.136022] ** kernel, report this immediately to your vendor! ** [ 0.137021] ** ** [ 0.138020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.139023] ************************************************************* [ 0.140996] NET: Registered protocol family 16 [ 0.141560] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.142083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.143159] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.145074] cpuidle: using governor menu [ 0.147364] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.150974] PCI: Using configuration type 1 for base access [ 0.154166] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.164071] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.169166] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.173212] cryptd: max_cpu_qlen set to 1000 [ 0.174321] ACPI: Added _OSI(Module Device) [ 0.175024] ACPI: Added _OSI(Processor Device) [ 0.176020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.178023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.184170] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.193866] ACPI: Interpreter enabled [ 0.195077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.197060] ACPI: Using IOAPIC for interrupt routing [ 0.199682] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.204552] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223134] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226139] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.230037] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234135] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.240897] acpiphp: Slot [2] registered [ 0.242249] acpiphp: Slot [5] registered [ 0.244218] acpiphp: Slot [6] registered [ 0.246209] acpiphp: Slot [7] registered [ 0.248205] acpiphp: Slot [8] registered [ 0.250199] acpiphp: Slot [9] registered [ 0.252311] acpiphp: Slot [10] registered [ 0.254279] acpiphp: Slot [3] registered [ 0.256155] acpiphp: Slot [4] registered [ 0.258169] acpiphp: Slot [11] registered [ 0.260184] acpiphp: Slot [12] registered [ 0.262159] acpiphp: Slot [13] registered [ 0.264191] acpiphp: Slot [14] registered [ 0.266214] acpiphp: Slot [15] registered [ 0.268259] acpiphp: Slot [16] registered [ 0.270152] acpiphp: Slot [17] registered [ 0.272154] acpiphp: Slot [18] registered [ 0.274143] acpiphp: Slot [19] registered [ 0.276230] acpiphp: Slot [20] registered [ 0.277333] acpiphp: Slot [21] registered [ 0.279376] acpiphp: Slot [22] registered [ 0.281320] acpiphp: Slot [23] registered [ 0.283344] acpiphp: Slot [24] registered [ 0.285296] acpiphp: Slot [25] registered [ 0.288150] acpiphp: Slot [26] registered [ 0.289278] acpiphp: Slot [27] registered [ 0.291165] acpiphp: Slot [28] registered [ 0.293140] acpiphp: Slot [29] registered [ 0.295191] acpiphp: Slot [30] registered [ 0.297136] acpiphp: Slot [31] registered [ 0.299082] PCI host bridge to bus 0000:00 [ 0.301029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.303027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.306031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.309040] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.312032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.316038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.318170] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.321399] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.325898] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.340023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.346069] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.349029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.354027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.358022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.361475] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.365000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.368069] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372737] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.376020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.392021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.399024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405741] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.419024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.436036] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.461018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.477161] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.495019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.507026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.546022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.568949] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.579074] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.590018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.617020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.624000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.626030] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.635016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.656020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.668209] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.684045] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.699030] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.723019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.733796] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.740016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.760022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.815018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.829894] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.833380] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.836341] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.838444] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.841248] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.845270] iommu: Default domain type: Passthrough [ 0.846105] SCSI subsystem initialized [ 0.848209] ACPI: bus type USB registered [ 0.850918] usbcore: registered new interface driver usbfs [ 0.854162] usbcore: registered new interface driver hub [ 0.857149] usbcore: registered new device driver usb [ 0.862117] pps_core: LinuxPPS API ver. 1 registered [ 0.864019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.865000] PTP clock support registered [ 0.866043] EDAC MC: Ver: 3.0.0 [ 0.868358] PCI: Using ACPI for IRQ routing [ 0.870933] NetLabel: Initializing [ 0.871000] NetLabel: domain hash size = 128 [ 0.877020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.879103] NetLabel: unlabeled traffic allowed by default [ 0.881504] vgaarb: loaded [ 0.883243] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.884018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.894568] clocksource: Switched to clocksource kvm-clock [ 1.024178] VFS: Disk quotas dquot_6.6.0 [ 1.025934] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.028832] *** VALIDATE ramfs *** [ 1.030394] *** VALIDATE hugetlbfs *** [ 1.032295] pnp: PnP ACPI init [ 1.035058] pnp: PnP ACPI: found 6 devices [ 1.062697] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.066542] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.068996] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.071490] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.074097] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.076704] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.080122] NET: Registered protocol family 2 [ 1.087720] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.186461] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.190982] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.198129] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.201858] TCP: Hash tables configured (established 65536 bind 65536) [ 1.205291] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.208795] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.212329] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.215599] NET: Registered protocol family 1 [ 1.218761] RPC: Registered named UNIX socket transport module. [ 1.221752] RPC: Registered udp transport module. [ 1.223953] RPC: Registered tcp transport module. [ 1.225378] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.228280] NET: Registered protocol family 44 [ 1.230340] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.232820] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.235462] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.238362] PCI: CLS 0 bytes, default 64 [ 1.240449] Unpacking initramfs... [ 3.131273] debug: unmapping init [mem 0xffff8d2c3cc54000-0xffff8d2c3ffbffff] [ 3.139564] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.142441] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.147804] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.713291] Initialise system trusted keyrings [ 3.715184] Key type blacklist registered [ 3.717373] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.727305] zbud: loaded [ 3.730595] *** VALIDATE nfs *** [ 3.732102] *** VALIDATE nfs4 *** [ 3.733831] pstore: using deflate compression [ 3.738371] Platform Keyring initialized [ 3.873577] NET: Registered protocol family 38 [ 3.875695] Key type asymmetric registered [ 3.877647] Asymmetric key parser 'x509' registered [ 3.879944] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.883811] io scheduler mq-deadline registered [ 3.886190] io scheduler kyber registered [ 3.888334] io scheduler bfq registered [ 3.890470] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.894439] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.897937] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.901558] ACPI: Power Button [PWRF] [ 3.908183] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.917252] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.938802] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.948830] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.963139] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.992865] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.023218] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.028704] Non-volatile memory driver v1.3 [ 4.030574] Linux agpgart interface v0.103 [ 4.063748] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 4.066858] vda: detected capacity change from 0 to 74825728 [ 4.082276] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.085893] vdb: detected capacity change from 0 to 1073741824 [ 4.113337] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.116737] vdc: detected capacity change from 0 to 2621440000 [ 4.171618] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.174864] vdd: detected capacity change from 0 to 2621440000 [ 4.195591] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.199147] vde: detected capacity change from 0 to 4294967296 [ 4.219314] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.222695] vdf: detected capacity change from 0 to 4294967296 [ 4.230853] libphy: Fixed MDIO Bus: probed [ 4.252223] usbcore: registered new interface driver usbserial_generic [ 4.256417] usbserial: USB Serial support registered for generic [ 4.260838] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.293790] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.295657] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.310709] mousedev: PS/2 mouse device common for all mice [ 4.322051] rtc_cmos 00:05: RTC can wake from S4 [ 4.324811] rtc_cmos 00:05: registered as rtc0 [ 4.326696] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.327583] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.336326] intel_pstate: CPU model not supported [ 4.338701] hid: raw HID events driver (C) Jiri Kosina [ 4.345205] usbcore: registered new interface driver usbhid [ 4.347072] usbhid: USB HID core driver [ 4.348931] drop_monitor: Initializing network drop monitor service [ 4.351448] Initializing XFRM netlink socket [ 4.354394] NET: Registered protocol family 10 [ 4.359364] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.361542] Segment Routing with IPv6 [ 4.369294] NET: Registered protocol family 17 [ 4.371287] mpls_gso: MPLS GSO support [ 4.377125] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.384197] RAS: Correctable Errors collector initialized. [ 4.386808] AVX version of gcm_enc/dec engaged. [ 4.388680] AES CTR mode by8 optimization enabled [ 4.503921] sched_clock: Marking stable (4503898472, 0)->(5818717243, -1314818771) [ 4.509988] registered taskstats version 1 [ 4.512744] Loading compiled-in X.509 certificates [ 4.515292] zswap: loaded using pool lzo/zbud [ 4.543186] Key type big_key registered [ 4.555377] Key type encrypted registered [ 4.556482] ima: No TPM chip found, activating TPM-bypass! [ 4.557816] ima: Allocated hash algorithm: sha1 [ 4.559221] ima: No architecture policies found [ 4.560564] evm: Initialising EVM extended attributes: [ 4.561713] evm: security.selinux [ 4.562427] evm: security.ima [ 4.562978] evm: security.capability [ 4.563923] evm: HMAC attrs: 0x1 [ 4.565903] rtc_cmos 00:05: setting system clock to 2026-08-22 20:56:30 UTC (1787432190) [ 4.570909] debug: unmapping init [mem 0xffffffff93203000-0xffffffff933fffff] [ 4.573169] debug: unmapping init [mem 0xffffffff91f82000-0xffffffff92258fff] [ 4.581198] Write protecting the kernel read-only data: 28672k [ 4.586610] debug: unmapping init [mem 0xffffffff90603000-0xffffffff907fffff] [ 4.590501] debug: unmapping init [mem 0xffffffff90f14000-0xffffffff90ffffff] [ 4.647695] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.658955] systemd[1]: Detected virtualization kvm. [ 4.661676] systemd[1]: Detected architecture x86-64. [ 4.664022] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.695281] systemd[1]: No hostname configured. [ 4.697142] systemd[1]: Set hostname to . [ 4.699438] random: systemd: uninitialized urandom read (16 bytes read) [ 4.701949] systemd[1]: Initializing machine ID from random generator. [ 4.792995] random: ln: uninitialized urandom read (6 bytes read) [ 4.906732] random: systemd: uninitialized urandom read (16 bytes read) [ 4.909226] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.913741] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.922837] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ 5.625076] hrtimer: interrupt took 2990494 ns [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.857294] device-mapper: uevent: version 1.0.3 [ 6.859210] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.033328] virtio_net virtio0 ens2: renamed from eth0 [ 9.497009] random: fast init done [ 9.941461] scsi host0: ata_piix [ 9.958146] scsi host1: ata_piix [ 10.259882] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.261916] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.077625] random: crng init done [ 16.137246] random: 7 urandom warning(s) missed due to ratelimiting [ 19.532123] dracut-initqueue[603]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.400063] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.890705] printk: systemd: 26 output lines suppressed due to ratelimiting [ 27.283729] SELinux: Disabled at runtime. [ 27.493770] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.515735] systemd[1]: Detected virtualization kvm. [ 27.523506] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 30.328312] systemd[1]: initrd-switch-root.service: Succeeded. [ 30.343585] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 30.380212] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 30.389202] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 30.409915] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 30.469888] systemd[1]: Starting Journal Service... Starting Journal Service... [ 30.525474] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 30.938538] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 33.339680] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 35.054274] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 35.086662] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 35.322395] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.482760] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 43.659379] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (13s / no limit)[ 44.200492] NFS: Registering the id_resolver key type [ 44.202736] Key type id_resolver registered [ 44.208521] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg237-server login: [ 114.716425] libcfs: loading out-of-tree module taints kernel. [ 114.874095] Key type ._llcrypt registered [ 114.875687] Key type .llcrypt registered [ 115.122345] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 134.276766] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 135.717239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 135.737384] alg: No test for adler32 (adler32-zlib) [ 137.443034] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 138.470127] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 140.400895] Key type lgssc registered [ 142.084559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 159.975510] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 211.010720] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 227.603917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 227.642876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 229.030191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 229.074811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 229.217158] Lustre: lustre-MDT0000: new disk, initializing [ 229.278488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.294741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 234.457717] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.752245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 248.953759] Lustre: 6531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 248.992835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 248.999576] Lustre: Skipped 1 previous similar message [ 249.092245] Lustre: lustre-MDT0001: new disk, initializing [ 249.170306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 249.210452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 249.239134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 254.368811] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.592444] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 269.399970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 269.808764] Lustre: lustre-OST0000: new disk, initializing [ 269.813616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 269.822871] Lustre: 8470:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 269.892544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 274.987583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 274.999388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 275.055456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 278.375485] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 295.498578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 295.717085] Lustre: lustre-OST0001: new disk, initializing [ 295.729551] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 295.739969] Lustre: 9556:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 295.826153] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 302.628309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 302.647522] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 302.776337] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 302.792363] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 315.829085] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 324.750482] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 334.068788] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing check_logdir /tmp/testlogs/ [ 342.056621] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing yml_node [ 348.657691] Lustre: DEBUG MARKER: Client: 2.17.57.63 [ 352.926588] Lustre: DEBUG MARKER: MDS: 2.17.57.63 [ 355.895775] Lustre: DEBUG MARKER: OSS: 2.17.57.63 [ 358.427874] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 17:02:23 EDT 2026 [ 378.501427] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 380.712171] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 397.804707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 397.825623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 397.851035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 399.841604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 399.845564] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.887258] Lustre: Skipped 2 previous similar messages [ 404.961626] LustreError: 6536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 405.004049] LustreError: 6536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 405.609704] Lustre: server umount lustre-MDT0000 complete [ 409.619935] LustreError: 7486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787432595 with bad export cookie 14835384409405608899 [ 409.630635] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 409.635685] LustreError: 7486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 410.092635] LustreError: 8969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.094887] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.095188] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 410.095201] Lustre: Skipped 3 previous similar messages [ 410.133244] LustreError: 8969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 410.160090] Lustre: Skipped 1 previous similar message [ 414.120125] LustreError: 6537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 414.120242] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 414.174706] Lustre: Skipped 2 previous similar messages [ 416.190791] Lustre: server umount lustre-MDT0001 complete [ 420.692807] Lustre: server umount lustre-OST0000 complete [ 424.331507] Lustre: server umount lustre-OST0001 complete [ 431.615253] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 442.679986] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 480.798376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.148255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 481.194355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 481.351595] Lustre: lustre-MDT0000: new disk, initializing [ 481.499829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.525354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 486.616793] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 498.382876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.550220] Lustre: 15112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 498.622320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 498.629246] Lustre: Skipped 1 previous similar message [ 498.741370] Lustre: lustre-MDT0001: new disk, initializing [ 498.860312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 498.913278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 498.929475] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 504.647793] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 515.846871] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 522.081532] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 529.491571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 529.839761] Lustre: lustre-OST0000: new disk, initializing [ 529.844532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 529.850918] Lustre: 17083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 529.959808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.515544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 531.523423] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 531.662939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 537.194083] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 547.186943] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 552.111474] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 552.228812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 558.425383] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 558.635389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 567.788938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.792217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 567.792836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 567.817033] Lustre: Skipped 1 previous similar message [ 569.583827] Lustre: server umount lustre-OST0000 complete [ 575.971594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 575.987560] LustreError: Skipped 1 previous similar message [ 576.002759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 576.029695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 576.044371] Lustre: Skipped 1 previous similar message [ 588.255396] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 588.731809] Lustre: server umount lustre-MDT0000 complete [ 592.368698] LustreError: 15125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.394270] LustreError: 15125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 593.343448] LustreError: 15105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787432779 with bad export cookie 14835384409405610439 [ 593.352339] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 593.364427] LustreError: 15105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 593.732604] Lustre: server umount lustre-MDT0001 complete [ 597.911063] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 17:06:22 (1787432782) [ 606.247856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 606.733324] LustreError: 19348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.830768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 612.146818] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 616.419502] LustreError: 19349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.434917] LustreError: 19349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 621.206996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.536742] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 621.546883] Lustre: Skipped 6 previous similar messages [ 626.546434] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 634.042712] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 639.859232] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 647.300852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 655.556907] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 663.069761] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 674.797329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 674.805921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 674.814709] Lustre: Skipped 1 previous similar message [ 674.818479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 680.483519] Lustre: server umount lustre-OST0000 complete [ 691.702583] Lustre: server umount lustre-MDT0000 complete [ 692.195475] LustreError: 19344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 692.225332] LustreError: 19344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 696.290228] LustreError: 19328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787432882 with bad export cookie 14835384409405611321 [ 696.295391] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 696.308729] LustreError: 19328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 696.744804] Lustre: server umount lustre-MDT0001 complete [ 710.778549] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 714.195336] Key type lgssc unregistered [ 714.699250] LNet: 22887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 714.708217] LNetError: 22887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 714.735234] LNet: Removed LNI 192.168.202.137@tcp [ 715.838358] Key type .llcrypt unregistered [ 715.840752] Key type ._llcrypt unregistered [ 743.421207] Key type ._llcrypt registered [ 743.424588] Key type .llcrypt registered [ 743.721860] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 745.540518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 745.588982] alg: No test for adler32 (adler32-zlib) [ 746.805825] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 747.107221] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 748.783262] Key type lgssc registered [ 750.188416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 760.202512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 760.234071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 761.832430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 767.315566] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 772.084934] LustreError: 24299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 772.105508] LustreError: 24299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 777.187428] LustreError: 24300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 777.472427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 777.956502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 782.633713] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 790.136563] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 796.889453] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 804.809737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 805.138486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.264329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 812.822678] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 820.801047] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 838.115561] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 838.122289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.139575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 838.173535] Lustre: Skipped 1 previous similar message [ 840.340576] Lustre: server umount lustre-OST0000 complete [ 844.771592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 844.783713] LustreError: Skipped 1 previous similar message [ 844.790903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.808635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 844.810431] Lustre: Skipped 1 previous similar message [ 848.354135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 848.360884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 848.410375] Lustre: Skipped 1 previous similar message [ 853.472067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 853.474890] Lustre: Skipped 1 previous similar message [ 858.607993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 859.103738] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 859.546745] Lustre: server umount lustre-MDT0000 complete [ 863.717060] LustreError: 24299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 863.744573] LustreError: 24299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 863.916208] LustreError: 24280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787433049 with bad export cookie 3918029314198497749 [ 863.917996] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 863.937396] LustreError: 24280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 864.206437] Lustre: server umount lustre-MDT0001 complete [ 878.541705] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 882.628535] Key type lgssc unregistered [ 882.925987] LNet: 28285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.940340] LNetError: 28285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.967612] LNet: Removed LNI 192.168.202.137@tcp [ 883.910342] Key type .llcrypt unregistered [ 883.912861] Key type ._llcrypt unregistered [ 907.259243] Key type ._llcrypt registered [ 907.264571] Key type .llcrypt registered [ 907.428986] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 908.890559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 909.078284] alg: No test for adler32 (adler32-zlib) [ 910.174299] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 910.535445] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 912.287412] Key type lgssc registered [ 913.858275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 922.058208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 922.071229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 923.440056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.047934] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 933.861793] LustreError: 29699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 933.889159] LustreError: 29699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 936.681375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.062168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 942.537424] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 950.546675] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 956.735881] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 970.531707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 971.065687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 977.947255] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 985.909192] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 993.672201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1006.579969] Lustre: server umount lustre-OST0000 complete [ 1013.731292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1013.746352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.765680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1019.362115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1019.364731] Lustre: Skipped 2 previous similar messages [ 1020.502234] Lustre: server umount lustre-MDT0000 complete [ 1023.393899] Lustre: 29012:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787433193/real 1787433193] req@00000000f061962e x1874259446104192/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787433209 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1023.430888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.447689] Lustre: Skipped 1 previous similar message [ 1023.459290] LustreError: 30646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1023.482643] LustreError: 30646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1024.202840] LustreError: 29679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787433210 with bad export cookie 10492756349310257524 [ 1024.221873] LustreError: 29679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1024.224645] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1024.518471] Lustre: server umount lustre-MDT0001 complete [ 1036.616732] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1041.425585] Key type lgssc unregistered [ 1041.953882] LNet: 33604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.970762] LNetError: 33604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.006773] LNet: Removed LNI 192.168.202.137@tcp [ 1043.391582] Key type .llcrypt unregistered [ 1043.397726] Key type ._llcrypt unregistered [ 1069.810301] Key type ._llcrypt registered [ 1069.812724] Key type .llcrypt registered [ 1069.925505] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1071.585306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1071.675728] alg: No test for adler32 (adler32-zlib) [ 1072.879019] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1073.229371] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1075.023481] Key type lgssc registered [ 1076.217719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1084.726230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1084.743408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.299239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.989774] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1096.678436] LustreError: 35027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1096.701659] LustreError: 35027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1100.211796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.641422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1105.132361] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1112.565608] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1118.062257] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1123.956548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1124.249931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1130.162177] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1131.438498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1137.631528] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1146.850794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1146.852678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.861067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1146.866549] Lustre: Skipped 1 previous similar message [ 1151.970562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1151.979458] Lustre: Skipped 2 previous similar messages [ 1152.913275] Lustre: server umount lustre-OST0000 complete [ 1157.089968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1157.094449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.102534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1157.153377] Lustre: Skipped 1 previous similar message [ 1162.212635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1162.223148] Lustre: Skipped 2 previous similar messages [ 1162.910671] Lustre: server umount lustre-MDT0000 complete [ 1167.268533] LustreError: 35007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787433353 with bad export cookie 9313371348095794561 [ 1167.276853] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1167.285203] LustreError: 35007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1167.706571] Lustre: server umount lustre-MDT0001 complete [ 1179.895781] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1182.845706] Key type lgssc unregistered [ 1183.225684] LNet: 38562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.241780] LNetError: 38562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.266329] LNet: Removed LNI 192.168.202.137@tcp [ 1184.176716] Key type .llcrypt unregistered [ 1184.179790] Key type ._llcrypt unregistered [ 1207.163586] Key type ._llcrypt registered [ 1207.166278] Key type .llcrypt registered [ 1207.268762] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1208.788018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1208.908608] alg: No test for adler32 (adler32-zlib) [ 1210.024339] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1210.311413] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1211.983176] Key type lgssc registered [ 1213.173622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1220.018624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1220.043900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.345229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.227496] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1231.841955] LustreError: 39976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.853872] LustreError: 39976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1232.960623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1233.271922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1237.947273] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1244.242948] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1249.940599] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1257.255703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1257.619157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.815478] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1266.861869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1271.145108] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1278.969787] Lustre: lustre-OST0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 1279.460530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1279.470706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1279.481711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1282.529798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1282.535244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1282.550892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1287.660204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1287.664507] Lustre: Skipped 3 previous similar messages [ 1292.256260] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1292.605078] Lustre: server umount lustre-OST0000 complete [ 1311.971107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.996347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1311.998669] Lustre: Skipped 1 previous similar message [ 1318.437396] Lustre: server umount lustre-MDT0000 complete [ 1322.387653] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787433508 with bad export cookie 9848744131884879497 [ 1322.399266] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1322.404471] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1328.877742] Lustre: server umount lustre-MDT0001 complete [ 1338.996691] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1341.530897] Key type lgssc unregistered [ 1341.809689] LNet: 43691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1341.817580] LNetError: 43691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1341.846710] LNet: Removed LNI 192.168.202.137@tcp [ 1342.800362] Key type .llcrypt unregistered [ 1342.803026] Key type ._llcrypt unregistered [ 1367.046294] Key type ._llcrypt registered [ 1367.048939] Key type .llcrypt registered [ 1367.147720] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1368.216748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1368.330142] alg: No test for adler32 (adler32-zlib) [ 1369.547462] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1369.762972] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1371.415206] Key type lgssc registered [ 1372.616524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1381.364636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1381.379396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.849412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.872599] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1391.014571] LustreError: 45104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.032580] LustreError: 45104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1393.070566] LustreError: 45105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.893635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.156318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1401.519587] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1409.219409] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1415.145632] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1421.322235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1421.672207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1428.535232] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1428.874064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1436.498628] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1443.300074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1443.332886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.354210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1444.330269] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.341775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1444.365539] Lustre: Skipped 3 previous similar messages [ 1445.401455] Lustre: server umount lustre-MDT0000 complete [ 1448.742995] LustreError: 46948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787433634 with bad export cookie 7447324596712607182 [ 1448.754288] LustreError: 46948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1448.766798] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1449.407931] LustreError: 46539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.441438] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.449676] LustreError: 46539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1449.488546] Lustre: Skipped 1 previous similar message [ 1449.500138] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1453.472486] LustreError: 46539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1453.472503] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1453.498634] Lustre: Skipped 1 previous similar message [ 1458.594409] LustreError: 46539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1458.594808] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1458.611348] LustreError: 46539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1458.621757] Lustre: Skipped 1 previous similar message [ 1463.263257] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1463.586660] Lustre: server umount lustre-MDT0001 complete [ 1479.186924] Lustre: server umount lustre-OST0000 complete [ 1496.200119] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1499.114380] Key type lgssc unregistered [ 1499.466914] LNet: 48970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.478105] LNetError: 48970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.512070] LNet: Removed LNI 192.168.202.137@tcp [ 1500.579798] Key type .llcrypt unregistered [ 1500.587417] Key type ._llcrypt unregistered [ 1523.809970] Key type ._llcrypt registered [ 1523.812819] Key type .llcrypt registered [ 1523.932061] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1525.473489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1525.713090] alg: No test for adler32 (adler32-zlib) [ 1526.797795] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1527.208988] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1529.063596] Key type lgssc registered [ 1530.350462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1537.467420] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1537.479257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1553.055995] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1558.177691] LustreError: 50355:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [ 1584.863704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1590.615345] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1597.727851] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1764.432821] Lustre: server umount lustre-OST0000 complete [ 1778.987567] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1781.833897] Key type lgssc unregistered [ 1782.104668] LNet: 52034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.115529] LNetError: 52034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.136238] LNet: Removed LNI 192.168.202.137@tcp [ 1782.814557] Key type .llcrypt unregistered [ 1782.816315] Key type ._llcrypt unregistered [ 1802.908208] Key type ._llcrypt registered [ 1802.910369] Key type .llcrypt registered [ 1803.006373] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1804.175877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1804.205037] alg: No test for adler32 (adler32-zlib) [ 1805.245439] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1805.499690] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1807.151296] Key type lgssc registered [ 1808.060418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1814.355672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1814.382946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.896849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.912385] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1825.262306] LustreError: 53451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1825.278350] LustreError: 53451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1828.395182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.729380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1833.183965] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1841.522413] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1847.828585] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1855.529130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1855.970283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.105899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1862.362408] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1871.442646] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1874.658097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1874.660567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1879.920125] Lustre: server umount lustre-OST0000 complete [ 1885.152801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1885.154399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.169291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1885.169300] Lustre: Skipped 1 previous similar message [ 1885.221936] Lustre: Skipped 1 previous similar message [ 1890.293833] LustreError: 54465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1890.452620] Lustre: server umount lustre-MDT0000 complete [ 1893.807065] LustreError: 53432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787434079 with bad export cookie 5298842021092723184 [ 1893.811772] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1893.814082] LustreError: 53432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1894.161496] Lustre: server umount lustre-MDT0001 complete [ 1904.663385] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1907.042305] Key type lgssc unregistered [ 1907.264586] LNet: 56984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1907.271386] LNetError: 56984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1907.294152] LNet: Removed LNI 192.168.202.137@tcp [ 1908.208194] Key type .llcrypt unregistered [ 1908.210254] Key type ._llcrypt unregistered [ 1928.251930] Key type ._llcrypt registered [ 1928.255236] Key type .llcrypt registered [ 1928.366733] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1929.707409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1929.719538] alg: No test for adler32 (adler32-zlib) [ 1930.755477] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1930.981300] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1932.647312] Key type lgssc registered [ 1933.706355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1941.729639] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1941.758412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1957.344131] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1962.463865] LustreError: 58351:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [ 1989.156077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.812732] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2002.603918] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2009.474497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.970944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.022975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2014.292446] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2015.207281] LustreError: 58376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.240677] LustreError: 58376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2020.331493] LustreError: 58377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2022.671697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.099405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2027.481896] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2034.878217] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2041.300403] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2043.876815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2043.890187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.899336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2044.900895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2044.912878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.690214] Lustre: server umount lustre-OST0000 complete [ 2074.595980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.614191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2074.614795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2074.614799] Lustre: Skipped 1 previous similar message [ 2074.625929] Lustre: Skipped 1 previous similar message [ 2079.712951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2079.718722] Lustre: Skipped 1 previous similar message [ 2084.833496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2087.391239] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2087.719206] Lustre: server umount lustre-MDT0000 complete [ 2089.963218] LustreError: 59387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.995492] LustreError: 59387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2091.993335] LustreError: 58357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787434277 with bad export cookie 8410613967950257741 [ 2092.005084] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2106.342871] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2106.699543] Lustre: server umount lustre-MDT0001 complete [ 2116.973539] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2120.069932] Key type lgssc unregistered [ 2120.421294] LNet: 62101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.434366] LNetError: 62101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.456710] LNet: Removed LNI 192.168.202.137@tcp [ 2121.231377] Key type .llcrypt unregistered [ 2121.233668] Key type ._llcrypt unregistered [ 2143.953315] Key type ._llcrypt registered [ 2143.954578] Key type .llcrypt registered [ 2144.008772] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2145.337843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2145.370279] alg: No test for adler32 (adler32-zlib) [ 2146.523736] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2146.814208] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2148.519149] Key type lgssc registered [ 2149.717694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2157.877951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2157.910550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2159.336770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.472923] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2169.828102] LustreError: 63523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.849891] LustreError: 63523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2171.668485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2172.036062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2178.695236] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2186.881826] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2192.944542] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2199.650149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2199.893979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.048494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2206.054662] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2216.063910] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2235.803363] Lustre: server umount lustre-OST0000 complete [ 2242.020360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.023918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.035743] Lustre: Skipped 1 previous similar message [ 2247.138631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.147826] Lustre: Skipped 1 previous similar message [ 2249.320928] Lustre: server umount lustre-MDT0000 complete [ 2252.257585] Lustre: 62833:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787434422/real 1787434422] req@0000000038081869 x1874260742143744/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787434438 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2252.264032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.267828] LustreError: 63517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.289456] Lustre: 62833:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2253.691336] LustreError: 63504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787434439 with bad export cookie 10191377885507372743 [ 2253.698965] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2253.728145] LustreError: 63504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2254.375211] Lustre: server umount lustre-MDT0001 complete [ 2265.957161] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2268.771905] Key type lgssc unregistered [ 2269.100158] LNet: 67081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2269.112959] LNetError: 67081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2269.146827] LNet: Removed LNI 192.168.202.137@tcp [ 2270.104764] Key type .llcrypt unregistered [ 2270.107921] Key type ._llcrypt unregistered [ 2295.860640] Key type ._llcrypt registered [ 2295.862864] Key type .llcrypt registered [ 2295.994738] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2297.065944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2297.086479] alg: No test for adler32 (adler32-zlib) [ 2298.275406] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2298.552941] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2300.271198] Key type lgssc registered [ 2301.433806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2309.958686] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 17:34:54 (1787434494) [ 2316.431447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2316.444448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2317.899724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2321.492838] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2328.033292] LustreError: 68784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.050285] LustreError: 68784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2328.833994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2329.082065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2333.726117] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2342.825640] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2349.124446] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2356.045434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2356.372707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.022802] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2362.546037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2370.381570] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2380.240785] Lustre: Failing over lustre-MDT0000 [ 2380.269315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2380.279220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2380.293824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2380.494451] Lustre: server umount lustre-MDT0000 complete [ 2383.329431] LustreError: 68781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.352965] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2384.356694] LustreError: 69788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.365444] Lustre: Skipped 1 previous similar message [ 2384.377905] LustreError: 69788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2388.432187] LustreError: 68780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.306659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.403663] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2389.478941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2389.499322] Lustre: Skipped 2 previous similar messages [ 2389.525350] Lustre: *** cfs_fail_loc=135, val=0*** [ 2389.639638] LustreError: 71871:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2389.660559] LustreError: 71871:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.137@tcp: cfg command failed: rc = -2 [ 2389.674834] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2389.674834] [ 2389.688205] LustreError: MGC192.168.202.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2389.704229] LustreError: 71860:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2389.713385] LustreError: 71860:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2389.726518] LustreError: 71860:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2389.732657] LustreError: 71860:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2389.766259] Lustre: server umount lustre-MDT0000 complete [ 2389.775090] LustreError: 71860:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2393.535643] LustreError: 68780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.929487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.031819] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2396.049573] LustreError: 68775:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2396.297110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2396.346243] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2398.655583] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2401.165686] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2401.785298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2401.825061] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2401.881309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2412.002811] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.028984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2412.037233] Lustre: Skipped 1 previous similar message [ 2412.064700] Lustre: Skipped 1 previous similar message [ 2417.030858] Lustre: server umount lustre-OST0000 complete [ 2422.243190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2422.244488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.252512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2422.283176] Lustre: Skipped 1 previous similar message [ 2427.370622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2427.380881] Lustre: Skipped 2 previous similar messages [ 2427.713890] Lustre: server umount lustre-MDT0000 complete [ 2431.850986] LustreError: 69519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787434617 with bad export cookie 16782629704898480262 [ 2431.856738] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2432.024803] Lustre: server umount lustre-MDT0001 complete [ 2443.822774] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2446.699404] Key type lgssc unregistered [ 2447.032365] LNet: 73653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2447.043164] LNetError: 73653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2448.102767] LNet: Removed LNI 192.168.202.137@tcp [ 2448.996256] Key type .llcrypt unregistered [ 2449.002500] Key type ._llcrypt unregistered [ 2472.240671] Key type ._llcrypt registered [ 2472.242490] Key type .llcrypt registered [ 2472.374890] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2473.288702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2473.340864] alg: No test for adler32 (adler32-zlib) [ 2474.446685] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2474.649867] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2476.335168] Key type lgssc registered [ 2477.305322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2484.277494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2484.295680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2485.714951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2489.883855] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2495.980872] LustreError: 75065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2495.994475] LustreError: 75065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2497.491143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.900874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2503.035707] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2510.630470] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2516.087237] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2522.628801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2522.977980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2530.118040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2531.013857] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2539.757620] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2549.563353] Lustre: Failing over lustre-MDT0000 [ 2549.889553] Lustre: server umount lustre-MDT0000 complete [ 2552.790693] LustreError: 75059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.803107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2552.829746] Lustre: Skipped 2 previous similar messages [ 2557.891225] LustreError: 75062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.925114] LustreError: 75062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2558.529795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2558.636133] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.787234] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2558.932818] LustreError: 78153:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2558.937839] LustreError: 78153:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.137@tcp: cfg command failed: rc = -9 [ 2558.944119] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2558.944119] [ 2558.952550] LustreError: MGC192.168.202.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2558.959685] LustreError: 78141:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2558.966249] LustreError: 78141:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2558.971879] LustreError: 78141:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2558.976835] LustreError: 78141:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2559.002773] Lustre: server umount lustre-MDT0000 complete [ 2559.010288] LustreError: 78141:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2563.021038] LustreError: 76501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.782323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.901605] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2566.166515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2566.256094] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2568.141407] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2570.616573] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2571.259409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2571.312185] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2571.395208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2581.505391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.522224] Lustre: Skipped 1 previous similar message [ 2581.527938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2586.607167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2586.628468] Lustre: Skipped 2 previous similar messages [ 2591.720135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2591.729984] Lustre: Skipped 1 previous similar message [ 2593.247139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2593.901872] Lustre: server umount lustre-OST0000 complete [ 2601.953101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2601.962300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.963614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2607.073484] LustreError: 75059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.094714] LustreError: 75059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2607.572631] Lustre: server umount lustre-MDT0000 complete [ 2611.407187] LustreError: 75047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787434797 with bad export cookie 12712353233776768939 [ 2611.408097] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2611.419493] LustreError: 75047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2611.640501] Lustre: server umount lustre-MDT0001 complete [ 2622.070188] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2625.181332] Key type lgssc unregistered [ 2625.572482] LNet: 79939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2625.580453] LNetError: 79939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2625.598705] LNet: Removed LNI 192.168.202.137@tcp [ 2626.557281] Key type .llcrypt unregistered [ 2626.560727] Key type ._llcrypt unregistered [ 2640.622868] Key type ._llcrypt registered [ 2640.624498] Key type .llcrypt registered [ 2640.714206] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2640.985323] Key type .llcrypt unregistered [ 2640.988728] Key type ._llcrypt unregistered [ 2664.295436] Key type ._llcrypt registered [ 2664.297641] Key type .llcrypt registered [ 2664.388777] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2665.124434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2665.153708] alg: No test for adler32 (adler32-zlib) [ 2666.180875] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2666.459665] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2668.128145] Key type lgssc registered [ 2668.866038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2676.768426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2676.803180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.472562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.706921] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2688.994807] LustreError: 81847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.022930] LustreError: 81847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2692.552544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2692.880237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2697.980423] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2707.141414] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2714.864929] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2721.638499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2721.999544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2724.073949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2727.513269] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2736.479074] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2747.892352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.910159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2747.918718] Lustre: Skipped 1 previous similar message [ 2751.772799] Lustre: server umount lustre-OST0000 complete [ 2758.112458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.126366] Lustre: Skipped 1 previous similar message [ 2758.133044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2761.874481] Lustre: server umount lustre-MDT0000 complete [ 2763.234360] LustreError: 81844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.255158] LustreError: 81844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2766.400876] LustreError: 81829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787434952 with bad export cookie 4316656196347431449 [ 2766.418510] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2766.419667] LustreError: 81829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2766.814691] Lustre: server umount lustre-MDT0001 complete [ 2776.343436] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2778.887544] Key type lgssc unregistered [ 2779.192892] LNet: 85385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2779.200882] LNetError: 85385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2779.234069] LNet: Removed LNI 192.168.202.137@tcp [ 2780.008880] Key type .llcrypt unregistered [ 2780.010691] Key type ._llcrypt unregistered [ 2803.594848] Key type ._llcrypt registered [ 2803.596199] Key type .llcrypt registered [ 2803.661709] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2804.722078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2804.953060] alg: No test for adler32 (adler32-zlib) [ 2806.002521] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2806.253750] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2807.903479] Key type lgssc registered [ 2809.810897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2820.563196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2820.634945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.082697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2827.691174] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2831.333355] LustreError: 86799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.349503] LustreError: 86799:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2836.453604] LustreError: 86798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.017433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.417379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2843.461112] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2851.762601] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2857.924491] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2863.662099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2864.155608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.230663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2870.076265] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2875.854859] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2886.624602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2886.635687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2886.646304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2887.648635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2887.654719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.655626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2892.141148] Lustre: server umount lustre-OST0000 complete [ 2897.888093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2897.891997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2897.897136] Lustre: Skipped 1 previous similar message [ 2897.901886] Lustre: Skipped 1 previous similar message [ 2901.974217] Lustre: server umount lustre-MDT0000 complete [ 2903.011136] LustreError: 86799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.029953] LustreError: 86799:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2905.617353] LustreError: 86779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787435091 with bad export cookie 489627170074612785 [ 2905.632154] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2906.097412] Lustre: server umount lustre-MDT0001 complete [ 2917.408461] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2920.850994] Key type lgssc unregistered [ 2921.265922] LNet: 90335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2921.272391] LNetError: 90335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2922.342433] LNet: Removed LNI 192.168.202.137@tcp [ 2923.252909] Key type .llcrypt unregistered [ 2923.257441] Key type ._llcrypt unregistered [ 2946.061189] Key type ._llcrypt registered [ 2946.063477] Key type .llcrypt registered [ 2946.202822] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2947.575192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2947.632556] alg: No test for adler32 (adler32-zlib) [ 2948.676205] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2948.854339] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2950.511171] Key type lgssc registered [ 2951.541786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2958.617727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2958.690229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2960.257132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.939559] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2970.596637] LustreError: 91754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2970.615199] LustreError: 91754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2973.270498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.766457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2978.262916] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2985.279801] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2990.455897] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2996.693062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2997.039769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.153680] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3005.252895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3009.866603] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3020.774421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.775980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3020.783771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3020.794496] Lustre: Skipped 1 previous similar message [ 3020.829022] Lustre: Skipped 1 previous similar message [ 3025.892653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3025.904205] Lustre: Skipped 1 previous similar message [ 3026.756507] Lustre: server umount lustre-OST0000 complete [ 3031.012470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.014293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3031.034077] Lustre: Skipped 1 previous similar message [ 3031.049878] Lustre: Skipped 1 previous similar message [ 3036.129234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3036.139977] Lustre: Skipped 1 previous similar message [ 3036.534409] Lustre: server umount lustre-MDT0000 complete [ 3040.116235] LustreError: 91734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787435226 with bad export cookie 6726813893805402976 [ 3040.118678] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3040.129989] LustreError: 91734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3040.421277] Lustre: server umount lustre-MDT0001 complete [ 3051.316046] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3054.131709] Key type lgssc unregistered [ 3054.341739] LNet: 95288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3054.356435] LNetError: 95288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3054.381033] LNet: Removed LNI 192.168.202.137@tcp [ 3055.264387] Key type .llcrypt unregistered [ 3055.268858] Key type ._llcrypt unregistered [ 3075.573808] Key type ._llcrypt registered [ 3075.576618] Key type .llcrypt registered [ 3075.683655] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3076.786823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3076.811744] alg: No test for adler32 (adler32-zlib) [ 3077.933493] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 3078.203188] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3079.919244] Key type lgssc registered [ 3081.108197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3089.615881] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3089.645764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3105.248629] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3110.367436] LustreError: 96654:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [ 3136.993701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3143.801949] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3150.798893] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3155.356596] Lustre: server umount lustre-OST0000 complete [ 3162.383406] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 17:49:08 (1787435348) [ 3167.840545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3168.225519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3172.732521] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3173.348503] LustreError: 98293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.357152] LustreError: 98293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3178.465658] LustreError: 98294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3181.071422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.615226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3185.866232] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3193.315727] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3199.106215] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3206.123955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3206.475637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.441953] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3213.684254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3220.126403] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3237.344692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3237.347904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3237.407621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3237.416111] Lustre: Skipped 1 previous similar message [ 3242.519709] Lustre: server umount lustre-OST0000 complete [ 3247.584394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.586482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.591403] Lustre: Skipped 2 previous similar messages [ 3247.597597] Lustre: Skipped 1 previous similar message [ 3252.708433] LustreError: 100406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.731791] LustreError: 100406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3252.811943] Lustre: server umount lustre-MDT0000 complete [ 3257.032179] LustreError: 99026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787435442 with bad export cookie 3906034746422049456 [ 3257.038681] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3257.041178] LustreError: 99026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3257.398338] Lustre: server umount lustre-MDT0001 complete [ 3270.725216] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3273.217372] Key type lgssc unregistered [ 3273.680651] LNet: 101991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3273.685199] LNetError: 101991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3273.706337] LNet: Removed LNI 192.168.202.137@tcp [ 3275.026178] Key type .llcrypt unregistered [ 3275.028109] Key type ._llcrypt unregistered [ 3322.374943] Key type ._llcrypt registered [ 3322.382342] Key type .llcrypt registered [ 3322.551170] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3324.280976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3324.375663] alg: No test for adler32 (adler32-zlib) [ 3325.741807] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 3326.136481] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3327.783765] Key type lgssc registered [ 3329.277950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3339.144201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3339.154059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.480691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3344.812747] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3351.018910] LustreError: 104179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3351.050146] LustreError: 104179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3353.286513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3353.588910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3358.211396] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3365.627357] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3371.203615] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3378.015419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3378.401077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.954335] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3385.607494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3393.130661] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3406.308228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3406.311430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.334668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3409.379774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3409.384397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3409.391798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3412.506136] Lustre: server umount lustre-OST0000 complete [ 3419.617132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.642213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3419.647441] Lustre: Skipped 2 previous similar messages [ 3422.825226] Lustre: server umount lustre-MDT0000 complete [ 3424.740712] LustreError: 106886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.390525] LustreError: 104158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787435612 with bad export cookie 6857693890656863640 [ 3426.409868] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3426.411612] LustreError: 104158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3426.954315] Lustre: server umount lustre-MDT0001 complete [ 3438.738748] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3441.469422] Key type lgssc unregistered [ 3441.964635] LNet: 107739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3441.969855] LNetError: 107739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3441.984607] LNet: Removed LNI 192.168.202.137@tcp [ 3443.220035] Key type .llcrypt unregistered [ 3443.223124] Key type ._llcrypt unregistered [ 3468.174648] Key type ._llcrypt registered [ 3468.183441] Key type .llcrypt registered [ 3468.331215] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3469.710429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3469.747966] alg: No test for adler32 (adler32-zlib) [ 3471.049737] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 3471.558531] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3473.351206] Key type lgssc registered [ 3474.814484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3483.883407] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3483.910529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3499.551777] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3504.671566] LustreError: 108986:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [ 3531.360439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3538.589492] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3547.092348] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3555.175202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.584175] LustreError: MGC192.168.202.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3555.607249] LustreError: 110011:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3555.617940] LustreError: 110011:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3555.639423] LustreError: 110011:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3555.825872] Lustre: server umount lustre-MDT0000 complete [ 3555.827893] LustreError: 110011:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3575.775142] Lustre: 108323:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787435745/real 1787435745] req@00000000ad0bea58 x1874262131477888/t0(0) o400->MGC192.168.202.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787435761 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3575.797368] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3575.806602] LustreError: 108998:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3575.961986] Lustre: server umount lustre-OST0000 complete [ 3588.718117] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 3599.900001] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3639.082287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3639.389938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3639.431474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3639.563493] Lustre: lustre-MDT0000: new disk, initializing [ 3639.701112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3639.720848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3645.448542] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3656.595715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3656.788984] Lustre: 112806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3656.883992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3656.886890] Lustre: Skipped 1 previous similar message [ 3656.943260] Lustre: lustre-MDT0001: new disk, initializing [ 3657.033635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3657.083684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3657.098169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3661.955737] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3672.414981] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3679.502059] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3686.632629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3687.077703] Lustre: lustre-OST0000: new disk, initializing [ 3687.088944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3687.099027] Lustre: 114778:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3687.226032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.425733] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3694.645904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3694.662693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3694.785796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3704.110413] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3709.713722] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3709.967636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3717.532461] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3717.953493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3725.795278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3725.799907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3725.804329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3725.811283] Lustre: Skipped 1 previous similar message [ 3730.919908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3730.941246] Lustre: Skipped 2 previous similar messages [ 3736.031138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3736.034727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3736.070806] Lustre: Skipped 1 previous similar message [ 3736.301497] Lustre: server umount lustre-OST0000 complete [ 3741.163974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3741.178214] Lustre: Skipped 1 previous similar message [ 3741.189591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3741.204278] Lustre: Skipped 1 previous similar message [ 3746.272757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3746.990319] Lustre: server umount lustre-MDT0000 complete [ 3751.453520] LustreError: 112797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787435937 with bad export cookie 7969846724080589226 [ 3751.458049] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3751.463823] LustreError: 112797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3752.123387] Lustre: server umount lustre-MDT0001 complete [ 3763.404425] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 17:59:07 (1787435947) [ 3768.187144] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3786.733323] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 3797.825803] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3833.850099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3834.261869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3834.292992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3834.376459] Lustre: lustre-MDT0000: new disk, initializing [ 3834.433551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3834.454794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3840.074369] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3850.941191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3851.036542] Lustre: 119680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3851.090214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3851.093582] Lustre: Skipped 1 previous similar message [ 3851.156457] Lustre: lustre-MDT0001: new disk, initializing [ 3851.254602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3851.301407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3851.316929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3856.234363] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3868.316623] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3877.647990] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3888.226749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3888.642570] Lustre: lustre-OST0000: new disk, initializing [ 3888.647416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3888.660192] Lustre: 121659:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3888.786114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3895.899150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3895.905992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3896.072934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3896.279781] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3909.780586] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3917.398399] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3917.675232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3924.504373] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3924.798898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3932.131688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3932.157263] LustreError: Skipped 1 previous similar message [ 3932.162053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.185919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3932.204921] Lustre: Skipped 2 previous similar messages [ 3934.052651] Lustre: server umount lustre-OST0000 complete [ 3942.369610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.395861] Lustre: Skipped 2 previous similar messages [ 3944.922857] Lustre: server umount lustre-MDT0000 complete [ 3947.491332] LustreError: 119692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3947.524843] LustreError: 119692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3948.440839] LustreError: 119672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787436134 with bad export cookie 7969846724080590108 [ 3948.464500] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3948.478693] LustreError: 119672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3948.969364] Lustre: server umount lustre-MDT0001 complete [ 3958.509599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3958.865945] LustreError: 123880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3958.946754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3964.285429] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3964.410031] LustreError: 123881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3968.488038] LustreError: 123880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.371928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3973.609809] LustreError: 123881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3979.277570] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3987.782779] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3996.903793] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4006.541505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4014.430938] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4022.901208] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4034.180830] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4039.136137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4039.143166] LustreError: Skipped 1 previous similar message [ 4039.149941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4039.168909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4039.186807] Lustre: Skipped 2 previous similar messages [ 4039.191287] Lustre: Skipped 1 previous similar message [ 4041.225466] Lustre: server umount lustre-OST0000 complete [ 4045.792673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4045.799764] LustreError: Skipped 1 previous similar message [ 4049.377402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4049.389380] Lustre: Skipped 1 previous similar message [ 4059.103453] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4059.503826] Lustre: server umount lustre-MDT0000 complete [ 4059.619848] LustreError: 123881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.356124] LustreError: 123862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787436250 with bad export cookie 7969846724080590990 [ 4064.397573] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4065.136753] Lustre: server umount lustre-MDT0001 complete [ 4078.890950] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4082.100923] Key type lgssc unregistered [ 4082.517268] LNet: 127514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4082.533260] LNetError: 127514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4082.554706] LNet: Removed LNI 192.168.202.137@tcp [ 4083.634202] Key type .llcrypt unregistered [ 4083.635774] Key type ._llcrypt unregistered [ 4104.531398] Key type ._llcrypt registered [ 4104.533021] Key type .llcrypt registered [ 4104.658462] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 4122.709567] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4123.262750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4123.325721] alg: No test for adler32 (adler32-zlib) [ 4124.281115] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 4124.474602] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4126.151254] Key type lgssc registered [ 4127.937852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4167.259498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4167.280610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4168.667378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4168.752533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4168.885671] Lustre: lustre-MDT0000: new disk, initializing [ 4169.026047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4169.048281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4174.865835] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4186.827879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.062485] Lustre: 130667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4187.143359] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4187.147786] Lustre: Skipped 1 previous similar message [ 4187.231316] Lustre: lustre-MDT0001: new disk, initializing [ 4187.340425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4187.402705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4187.425389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4192.506200] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4203.446643] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4212.123867] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4221.442402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4221.714275] Lustre: lustre-OST0000: new disk, initializing [ 4221.721152] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4221.735398] Lustre: 132639:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4221.865326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4229.277208] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4231.809260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4231.816258] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4231.951422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4239.001931] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4244.832174] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4245.139485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4250.081334] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4250.352306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4252.640436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4252.645794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.653091] LustreError: Skipped 1 previous similar message [ 4252.653812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4252.699325] Lustre: Skipped 1 previous similar message [ 4257.763082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4257.769906] Lustre: Skipped 2 previous similar messages [ 4258.599794] Lustre: server umount lustre-OST0000 complete [ 4264.417833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4264.427278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.459181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4268.013670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4268.024217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4268.044465] Lustre: Skipped 1 previous similar message [ 4269.358501] Lustre: server umount lustre-MDT0000 complete [ 4274.285070] LustreError: 130661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787436460 with bad export cookie 6295088935844684145 [ 4274.292834] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4274.302007] LustreError: 130661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4275.343617] Lustre: server umount lustre-MDT0001 complete [ 4284.165862] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:07:49 (1787436469) [ 4290.390063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4290.668878] LustreError: 135053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.679444] LustreError: 135053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4290.730329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4293.644411] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4296.160715] LustreError: 135054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4300.465974] LustreError: 135053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4304.292718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4304.870994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4313.331577] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4323.849650] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4332.739395] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4335.585713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4335.598116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4335.612326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4339.689508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4341.964974] Lustre: server umount lustre-MDT0000 complete [ 4344.802507] LustreError: 135050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.818234] LustreError: 135050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4346.294452] LustreError: 135035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787436532 with bad export cookie 6295088935844685027 [ 4346.318040] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4346.322084] LustreError: 135035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4346.651735] Lustre: server umount lustre-MDT0001 complete [ 4355.793216] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:09:00 (1787436540) [ 4363.154800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4379.104161] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4384.223901] LustreError: 137469:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [ 4409.952651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4415.912508] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4423.133831] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4425.046332] Lustre: server umount lustre-OST0000 complete [ 4436.024986] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:10:20 (1787436620) [ 4443.307969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4443.916955] LustreError: 138916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4444.037769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4449.296340] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4453.349323] LustreError: 138917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4453.359309] LustreError: 138917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4460.008658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4467.265292] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4480.754921] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4489.797891] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4497.849567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4498.335266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4498.349679] Lustre: Skipped 1 previous similar message [ 4505.660756] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4514.643966] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4524.000952] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4524.003747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4524.020446] Lustre: Skipped 1 previous similar message [ 4524.020923] Lustre: Skipped 2 previous similar messages [ 4530.046546] Lustre: server umount lustre-MDT0000 complete [ 4532.209317] LustreError: 138917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4532.259251] LustreError: 138917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4534.470703] LustreError: 140761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787436720 with bad export cookie 6295088935844685692 [ 4534.486520] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4534.502533] LustreError: 140761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4534.803627] Lustre: server umount lustre-MDT0001 complete [ 4548.568988] Lustre: server umount lustre-OST0000 complete [ 4557.752524] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:12:22 (1787436742) [ 4566.230917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.710507] LustreError: 142574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4566.883126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4575.067930] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4587.995283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4592.848488] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4601.848449] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4608.764710] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4616.473034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4623.434899] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4630.487974] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4635.889174] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4636.339225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4637.188633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4641.816571] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4642.014918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4643.808877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4643.818721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4643.837446] Lustre: Skipped 1 previous similar message [ 4643.846564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4643.856651] Lustre: Skipped 5 previous similar messages [ 4649.756956] Lustre: server umount lustre-OST0000 complete [ 4655.071912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4655.074727] LustreError: Skipped 1 previous similar message [ 4659.546950] Lustre: server umount lustre-MDT0000 complete [ 4662.241701] LustreError: 142574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4662.271468] LustreError: 142574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4663.256858] LustreError: 142554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787436849 with bad export cookie 6295088935844686763 [ 4663.279820] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4663.977285] Lustre: server umount lustre-MDT0001 complete [ 4674.023278] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:14:18 (1787436858) [ 4681.479364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4697.119897] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4702.239537] LustreError: 146739:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [ 4727.968124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4727.977781] Lustre: Skipped 2 previous similar messages [ 4735.209421] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4742.927387] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4750.113493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.134422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4758.009587] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4772.592431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4772.835426] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4772.838193] Lustre: Skipped 6 previous similar messages [ 4778.849661] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4787.829332] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4795.672588] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4801.120412] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4801.375736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4806.979637] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4807.388462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4813.793418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4813.800049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.814261] Lustre: Skipped 3 previous similar messages [ 4815.896757] Lustre: server umount lustre-MDT0000 complete [ 4819.438159] LustreError: 147775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4819.476156] LustreError: 147775:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 4819.661579] LustreError: 146745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787437005 with bad export cookie 6295088935844687785 [ 4819.677983] LustreError: 146745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4819.686748] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4820.056844] Lustre: server umount lustre-MDT0001 complete [ 4836.357064] Lustre: server umount lustre-OST0000 complete [ 4846.732574] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:17:11 (1787437031) [ 4853.256210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4868.771965] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4873.887666] LustreError: 150936:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [ 4907.532418] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4917.087241] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4924.479103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4924.934627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4930.451788] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4940.857860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4946.502947] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4953.425487] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4960.635764] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4966.904282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4967.127503] Lustre: lustre-OST0001: new disk, initializing [ 4967.133843] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4967.146963] Lustre: 153809:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4974.716855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4974.726715] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4974.935891] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4975.270320] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4986.750304] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4993.361627] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4993.950102] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5000.809902] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5001.210862] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5005.795065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5005.801811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5005.811592] Lustre: Skipped 3 previous similar messages [ 5005.822571] Lustre: Skipped 5 previous similar messages [ 5009.567987] Lustre: server umount lustre-OST0000 complete [ 5016.041910] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5016.052686] LustreError: Skipped 1 previous similar message [ 5019.607201] Lustre: server umount lustre-OST0001 complete [ 5030.005366] Lustre: server umount lustre-MDT0000 complete [ 5034.066494] LustreError: 150944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787437220 with bad export cookie 6295088935844688737 [ 5034.074429] LustreError: 150944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5034.081271] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5034.630836] Lustre: server umount lustre-MDT0001 complete [ 5059.527813] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:20:44 (1787437244) [ 5061.045880] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5062.655490] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5064.235080] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:20:49 (1787437249) [ 5070.783717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.100463] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5071.156234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5071.383316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5071.387603] Lustre: Skipped 6 previous similar messages [ 5075.662537] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5083.733139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5083.836924] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5083.863992] Lustre: 157369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5089.365842] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5098.528889] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5106.134701] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5115.380710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5115.549979] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5124.898305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5125.199430] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5132.248869] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5137.090337] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5137.233077] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5141.880216] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5142.387893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5150.691160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5150.696430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.697959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5150.709975] Lustre: Skipped 5 previous similar messages [ 5150.722283] Lustre: Skipped 5 previous similar messages [ 5152.021894] Lustre: server umount lustre-OST0000 complete [ 5175.006581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5182.417850] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5190.767545] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.791637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.796189] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5190.834369] Lustre: Skipped 1 previous similar message [ 5193.540563] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5201.857491] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5202.193440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5207.560549] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5207.825196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5214.602700] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff91b448f3c000.ost_server_uuid 50 [ 5215.983917] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91b448f3c000.ost_server_uuid in FULL state after 0 sec [ 5216.570619] LustreError: 157378:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.37@tcp) returned error from blocking AST (req@000000004ff49260 x1874262816474496 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000db68fa66/0x575ca608ca83bcbb lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.37@tcp remote: 0x6dc2711af20c3469 expref: 5 pid: 159422 timeout: 5316 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5216.605661] LustreError: lustre-MDT0000: A client on nid 192.168.202.37@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5216.617953] LustreError: 157366:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.37@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000db68fa66/0x575ca608ca83bcbb lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.37@tcp remote: 0x6dc2711af20c3469 expref: 6 pid: 159422 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5227.851307] Lustre: server umount lustre-OST0000 complete [ 5232.608328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5232.611192] LustreError: Skipped 1 previous similar message [ 5238.816809] Lustre: server umount lustre-MDT0000 complete [ 5242.336326] LustreError: 157382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5242.391322] LustreError: 157382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 5244.613727] LustreError: 157362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787437430 with bad export cookie 6295088935844689920 [ 5244.621791] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5244.623515] LustreError: 157362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5251.095499] Lustre: server umount lustre-MDT0001 complete [ 5267.224132] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5270.173682] Key type lgssc unregistered [ 5270.604940] LNet: 163486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5270.643455] LNetError: 163486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5270.658993] LNet: Removed LNI 192.168.202.137@tcp [ 5271.898981] Key type .llcrypt unregistered [ 5271.901162] Key type ._llcrypt unregistered [ 5301.792613] Key type ._llcrypt registered [ 5301.802604] Key type .llcrypt registered [ 5301.954086] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5303.605842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5303.649214] alg: No test for adler32 (adler32-zlib) [ 5304.640169] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 5304.770821] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 5306.407174] Key type lgssc registered [ 5307.649662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5319.535941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5319.547747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5321.191439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5327.415030] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5331.445996] LustreError: 164902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5331.476668] LustreError: 164902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5336.545633] LustreError: 164903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5337.190678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5337.520885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5342.182801] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5349.864574] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5355.945843] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5362.601695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5362.960711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5367.133990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5369.988059] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5377.564791] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5387.467737] Lustre: Failing over lustre-MDT0000 [ 5387.706719] Lustre: server umount lustre-MDT0000 complete [ 5387.753443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5387.769234] LustreError: 164897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5387.787588] LustreError: 164897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5392.877184] LustreError: 167377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5392.897491] LustreError: 167377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5397.986783] LustreError: 164897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5398.019674] LustreError: 164897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5399.916941] LustreError: 164885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.37@tcp arrived at 1787437585 with bad export cookie 6524050342657986043 [ 5407.750830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.832539] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5408.014176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5408.069719] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5412.948492] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5413.350716] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5413.369213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5415.461612] Lustre: lustre-MDT0000: Denying connection for new client 226915b8-d4c1-44b1-899d-8cbf91f90e28 (at 192.168.202.37@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5420.496404] Lustre: lustre-MDT0000: Denying connection for new client 226915b8-d4c1-44b1-899d-8cbf91f90e28 (at 192.168.202.37@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 5428.711398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5428.722080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5428.736087] Lustre: Skipped 3 previous similar messages [ 5430.447499] Lustre: server umount lustre-OST0000 complete [ 5434.071090] LustreError: 168633:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5434.086912] Lustre: 167981:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5434.103415] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5434.128078] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5434.140295] LustreError: 167981:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000086d4be90 x1874264053559296/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5434.182270] LustreError: 167981:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5434.194211] LustreError: 167981:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5434.210514] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5434.669246] Lustre: server umount lustre-MDT0000 complete [ 5438.953500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.972714] LustreError: 164898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.997169] LustreError: 164898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5439.290400] LustreError: 164883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787437625 with bad export cookie 6524050342657986148 [ 5439.296440] LustreError: 164883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5439.307220] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5439.321957] LustreError: 164212:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5445.675791] Lustre: server umount lustre-MDT0001 complete [ 5459.296305] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5463.196084] Key type lgssc unregistered [ 5463.504701] LNet: 169390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5463.521783] LNetError: 169390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5463.550821] LNet: Removed LNI 192.168.202.137@tcp [ 5464.849632] Key type .llcrypt unregistered [ 5464.851965] Key type ._llcrypt unregistered [ 5493.828826] Key type ._llcrypt registered [ 5493.836840] Key type .llcrypt registered [ 5494.006353] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5495.889364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5495.927359] alg: No test for adler32 (adler32-zlib) [ 5497.149581] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 5497.372965] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 5499.079865] Key type lgssc registered [ 5500.684848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5509.019976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5509.040622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5510.701684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.254449] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5520.873041] LustreError: 170809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5520.902915] LustreError: 170809:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5524.164410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5524.460176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5528.930918] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5536.008922] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5541.915863] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5548.374646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5548.799391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5554.915354] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5555.955073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5562.827434] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5570.535806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5570.548927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5570.566960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5571.555443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5571.556524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5571.559993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5571.577792] Lustre: Skipped 1 previous similar message [ 5574.156221] Lustre: server umount lustre-OST0000 complete [ 5580.768451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5580.778565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.793697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5585.073320] Lustre: server umount lustre-MDT0000 complete [ 5586.912628] LustreError: 170809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5586.942601] LustreError: 170809:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5588.819154] LustreError: 170790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787437774 with bad export cookie 14472704271253833485 [ 5588.825467] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5588.833553] LustreError: 170790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5589.097416] Lustre: server umount lustre-MDT0001 complete [ 5599.024793] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5601.634765] Key type lgssc unregistered [ 5601.944588] LNet: 174342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5601.953313] LNetError: 174342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5601.982562] LNet: Removed LNI 192.168.202.137@tcp [ 5602.900969] Key type .llcrypt unregistered [ 5602.902836] Key type ._llcrypt unregistered [ 5636.267950] Key type ._llcrypt registered [ 5636.269657] Key type .llcrypt registered [ 5636.380180] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5637.387431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5637.396128] alg: No test for adler32 (adler32-zlib) [ 5638.952233] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 5639.254109] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 5640.960181] Key type lgssc registered [ 5642.075850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5649.497879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5649.510098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5650.911197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5655.323853] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5660.133200] LustreError: 176049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5660.150031] LustreError: 176049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5664.127250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5664.546597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5668.536654] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5675.975657] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5681.333918] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5687.954883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5688.273177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5694.943533] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5696.575596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5702.838762] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5709.796210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5709.800104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5709.814696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5711.851309] LustreError: 178296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5711.852674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.896860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5711.985478] Lustre: server umount lustre-OST0000 complete [ 5716.961848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5716.985702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5716.986397] Lustre: Skipped 1 previous similar message [ 5717.005558] Lustre: Skipped 1 previous similar message [ 5722.081851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5722.091988] Lustre: Skipped 1 previous similar message [ 5722.325745] Lustre: server umount lustre-MDT0000 complete [ 5725.441254] LustreError: 178541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787437911 with bad export cookie 2716292921872922513 [ 5725.448039] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5725.453072] LustreError: 178541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5725.756751] Lustre: server umount lustre-MDT0001 complete [ 5736.037903] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5738.669837] Key type lgssc unregistered [ 5738.906117] LNet: 179583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5738.914886] LNetError: 179583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5738.943383] LNet: Removed LNI 192.168.202.137@tcp [ 5739.596849] Key type .llcrypt unregistered [ 5739.602569] Key type ._llcrypt unregistered [ 5761.555283] Key type ._llcrypt registered [ 5761.557313] Key type .llcrypt registered [ 5761.655921] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5762.678560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5762.861979] alg: No test for adler32 (adler32-zlib) [ 5764.098281] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 5764.385844] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 5766.103182] Key type lgssc registered [ 5767.362690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5778.900406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5778.920068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5780.663703] Lustre: *** cfs_fail_loc=135, val=0*** [ 5780.884843] LustreError: 181107:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5780.898744] LustreError: 181107:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.137@tcp: cfg command failed: rc = -2 [ 5780.930579] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5780.930579] [ 5780.966449] LustreError: MGC192.168.202.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5780.985270] LustreError: 181068:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5781.008490] LustreError: 181068:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5781.023557] LustreError: 181068:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5781.044626] LustreError: 181068:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5781.246086] Lustre: server umount lustre-MDT0000 complete [ 5781.249191] LustreError: 181068:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5789.476316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5789.926916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5795.113670] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5799.396980] LustreError: 181479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5799.408762] LustreError: 181479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5804.459358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5804.514045] LustreError: 181480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5804.791202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5809.531593] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5818.105464] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5823.270822] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5825.506050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5825.523570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5825.540928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.601179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.603703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5830.695614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5830.709552] Lustre: Skipped 2 previous similar messages [ 5835.745839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5835.753013] Lustre: Skipped 1 previous similar message [ 5839.327879] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5839.478104] Lustre: server umount lustre-MDT0000 complete [ 5840.867339] LustreError: 181475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5840.879726] LustreError: 181475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5842.345136] LustreError: 181461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787438028 with bad export cookie 10558488061508672323 [ 5842.359244] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5842.364538] LustreError: 181461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5842.603251] Lustre: server umount lustre-MDT0001 complete [ 5850.972781] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5853.313335] Key type lgssc unregistered [ 5853.611569] LNet: 183806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5853.616572] LNetError: 183806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5853.629402] LNet: Removed LNI 192.168.202.137@tcp [ 5854.241231] Key type .llcrypt unregistered [ 5854.242977] Key type ._llcrypt unregistered [ 5873.696594] Key type ._llcrypt registered [ 5873.699456] Key type .llcrypt registered [ 5873.851619] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5874.163722] Key type .llcrypt unregistered [ 5874.169882] Key type ._llcrypt unregistered [ 5896.902756] Key type ._llcrypt registered [ 5896.910847] Key type .llcrypt registered [ 5897.057947] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5898.509817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5898.570243] alg: No test for adler32 (adler32-zlib) [ 5899.673269] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 5899.937080] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 5901.624280] Key type lgssc registered [ 5903.098312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5915.453808] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5915.467157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5931.040417] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5936.162983] LustreError: 185944:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [ 5962.852575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5969.415665] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5978.564596] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5987.968889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5988.495345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5988.550030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5992.921443] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5993.965571] LustreError: 185969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5993.987405] LustreError: 185969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5998.060872] LustreError: 185970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5998.073654] LustreError: 185970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6001.928179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6002.399342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6007.347891] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6015.412907] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6021.000867] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6023.991556] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6037.986343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6038.013780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6038.051276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6043.621139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6043.623872] Lustre: Skipped 2 previous similar messages [ 6048.783967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6048.789277] Lustre: Skipped 1 previous similar message [ 6051.295426] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6051.430251] Lustre: server umount lustre-OST0000 complete [ 6058.464301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6058.475872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6058.486667] Lustre: Skipped 1 previous similar message [ 6058.489146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6058.492422] Lustre: Skipped 1 previous similar message [ 6060.885498] Lustre: server umount lustre-MDT0000 complete [ 6064.098110] LustreError: 188221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6064.135861] LustreError: 188221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6064.339705] LustreError: 185950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787438250 with bad export cookie 17993024831186640168 [ 6064.365097] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6064.722342] Lustre: server umount lustre-MDT0001 complete [ 6075.377185] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6079.071664] Key type lgssc unregistered [ 6079.530916] LNet: 189960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6079.539527] LNetError: 189960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6079.559611] LNet: Removed LNI 192.168.202.137@tcp [ 6080.676880] Key type .llcrypt unregistered [ 6080.679537] Key type ._llcrypt unregistered [ 6110.229757] Key type ._llcrypt registered [ 6110.231907] Key type .llcrypt registered [ 6110.331931] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6111.685953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6111.742290] alg: No test for adler32 (adler32-zlib) [ 6112.814580] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 6113.198824] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6114.943275] Key type lgssc registered [ 6115.700084] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6123.404419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6123.420485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6124.903635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6129.948778] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6134.243046] LustreError: 191374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6134.260240] LustreError: 191374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6137.997349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6138.450647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6142.660912] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6149.356378] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6156.043223] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6165.305830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6166.358546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6169.420970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6175.405391] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6185.622931] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6190.378349] Lustre: Failing over lustre-MDT0000 [ 6190.885214] Lustre: server umount lustre-MDT0000 complete [ 6192.601477] LustreError: 192968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6194.157503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6194.167141] LustreError: 191375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6194.191575] Lustre: Skipped 2 previous similar messages [ 6194.227969] LustreError: 191375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6197.695871] LustreError: 192968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6202.820541] LustreError: 192968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6202.836048] LustreError: 192968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6210.301279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.520979] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6210.529788] Lustre: 190687:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787438380/real 1787438380] req@000000003cd85d8b x1874264901839488/t0(0) o400->MGC192.168.202.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787438396 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6210.868549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6210.919751] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6212.996090] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6216.134091] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6216.198260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6216.213865] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6216.285226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6221.938121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6232.558433] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6244.846347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6244.860331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6244.893250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6245.177898] Lustre: server umount lustre-OST0000 complete [ 6252.013387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6252.022550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6252.028679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6252.042814] Lustre: Skipped 1 previous similar message [ 6252.056035] Lustre: Skipped 1 previous similar message [ 6255.663388] Lustre: server umount lustre-MDT0000 complete [ 6257.130196] LustreError: 193967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6257.203881] LustreError: 193967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6261.216697] Lustre: 190684:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787438431/real 1787438431] req@00000000f1343360 x1874264901872256/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787438447 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6261.247875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6261.407381] LustreError: 191355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787438447 with bad export cookie 13552550070297843307 [ 6261.416424] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6261.441241] LustreError: 191355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6261.960223] Lustre: server umount lustre-MDT0001 complete [ 6275.124373] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6277.266600] Key type lgssc unregistered [ 6277.587252] LNet: 196314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6277.596857] LNetError: 196314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6277.609874] LNet: Removed LNI 192.168.202.137@tcp [ 6278.270368] Key type .llcrypt unregistered [ 6278.271771] Key type ._llcrypt unregistered [ 6302.991336] Key type ._llcrypt registered [ 6302.993631] Key type .llcrypt registered [ 6303.081967] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6303.831696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6303.843151] alg: No test for adler32 (adler32-zlib) [ 6304.939376] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 6305.155165] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6306.863169] Key type lgssc registered [ 6308.130784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6317.055966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6317.103420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6318.672123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6323.336785] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6328.821641] LustreError: 197728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6328.842161] LustreError: 197728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6333.061654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.845754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6338.447515] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6347.429559] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6353.495440] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6361.052215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6361.463287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6363.533128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6368.281102] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6375.660766] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6379.534920] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6384.198430] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6405.440539] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6419.948077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6419.951436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6419.987727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6419.987971] LustreError: Skipped 1 previous similar message [ 6420.006611] Lustre: Skipped 1 previous similar message [ 6421.709581] Lustre: server umount lustre-OST0000 complete [ 6425.075794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6425.088439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6425.090703] Lustre: Skipped 2 previous similar messages [ 6425.108905] Lustre: Skipped 1 previous similar message [ 6430.177565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6430.194607] Lustre: Skipped 1 previous similar message [ 6435.296619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6435.308592] Lustre: Skipped 1 previous similar message [ 6439.391350] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6439.399476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6439.423670] Lustre: Skipped 1 previous similar message [ 6441.664569] Lustre: server umount lustre-MDT0000 complete [ 6444.512387] LustreError: 197729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6444.554456] LustreError: 197729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6446.179131] LustreError: 197708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787438632 with bad export cookie 11405510800032951247 [ 6446.185199] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6446.190731] LustreError: 197708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6446.829793] Lustre: server umount lustre-MDT0001 complete [ 6460.282943] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6462.973459] Key type lgssc unregistered [ 6463.307057] LNet: 201601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6463.315366] LNetError: 201601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6464.358305] LNet: Removed LNI 192.168.202.137@tcp [ 6465.437465] Key type .llcrypt unregistered [ 6465.442557] Key type ._llcrypt unregistered [ 6489.344674] Key type ._llcrypt registered [ 6489.347513] Key type .llcrypt registered [ 6489.467500] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6490.944429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6491.206519] alg: No test for adler32 (adler32-zlib) [ 6492.588562] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 6492.908619] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6494.599167] Key type lgssc registered [ 6495.879634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6504.201221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6504.217432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6505.637724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6510.303707] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6515.172173] LustreError: 202875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6515.190758] LustreError: 202875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6518.665635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6519.221270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6523.905071] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6531.743731] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6537.438801] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6544.386092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6544.649155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6550.841934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6550.854472] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6558.634933] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6565.317206] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6574.552267] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6584.034334] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6590.678206] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6602.214757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6602.227347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6602.257477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6604.264342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6604.267775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6604.301199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6607.859562] Lustre: server umount lustre-OST0000 complete [ 6614.497438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6614.516156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6614.523833] Lustre: Skipped 1 previous similar message [ 6619.617571] LustreError: 204656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6619.653221] LustreError: 204656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6620.453465] Lustre: server umount lustre-MDT0000 complete [ 6623.585656] LustreError: 205551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787438809 with bad export cookie 14586631934063197098 [ 6623.588523] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6623.593060] LustreError: 205551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6623.786375] Lustre: server umount lustre-MDT0001 complete [ 6633.880771] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6637.681377] Key type lgssc unregistered [ 6638.008204] LNet: 207851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6638.016992] LNetError: 207851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6639.088849] LNet: Removed LNI 192.168.202.137@tcp [ 6640.468730] Key type .llcrypt unregistered [ 6640.477464] Key type ._llcrypt unregistered [ 6664.497697] Key type ._llcrypt registered [ 6664.500131] Key type .llcrypt registered [ 6664.620977] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6666.161834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6666.214124] alg: No test for adler32 (adler32-zlib) [ 6667.396617] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 6667.677446] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6669.351263] Key type lgssc registered [ 6670.464856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6677.725169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6677.740740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.095248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6684.449830] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6689.251037] LustreError: 209243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6689.292990] LustreError: 209243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6693.484631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6694.107463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6700.726863] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6709.199484] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6716.560340] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6726.135553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6726.841683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6734.220787] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6741.694686] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6749.359644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6765.025467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6765.027646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6765.036277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6765.042549] Lustre: Skipped 1 previous similar message [ 6765.051393] Lustre: Skipped 1 previous similar message [ 6770.147133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6770.151521] Lustre: Skipped 1 previous similar message [ 6771.030933] Lustre: server umount lustre-OST0000 complete [ 6775.779349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6775.785630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6775.794200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6780.387629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6780.388159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6781.875102] Lustre: server umount lustre-MDT0000 complete [ 6785.505572] LustreError: 209239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6785.550962] LustreError: 209239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6787.585138] LustreError: 209224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787438973 with bad export cookie 15727098979078876410 [ 6787.594057] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6787.597149] LustreError: 209224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6787.981986] Lustre: server umount lustre-MDT0001 complete [ 6802.860387] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6806.237628] Key type lgssc unregistered [ 6806.643453] LNet: 212954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6806.655980] LNetError: 212954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6806.684976] LNet: Removed LNI 192.168.202.137@tcp [ 6808.034082] Key type .llcrypt unregistered [ 6808.038385] Key type ._llcrypt unregistered [ 6827.844198] Key type ._llcrypt registered [ 6827.846996] Key type .llcrypt registered [ 6827.955581] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6829.229403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6829.405379] alg: No test for adler32 (adler32-zlib) [ 6830.487362] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 6830.781296] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6832.495196] Key type lgssc registered [ 6834.020725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6843.604648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6843.619895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6845.264876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6849.387850] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6855.649766] LustreError: 214222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6855.662576] LustreError: 214222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6856.197263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6856.498363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6860.440737] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6867.085830] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6871.834946] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6879.764617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6880.498841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6887.953593] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6890.733723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6897.037986] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6912.481190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6912.490545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6912.502919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6916.578540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6916.585841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6916.586065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6916.596948] Lustre: Skipped 1 previous similar message [ 6917.830636] Lustre: server umount lustre-OST0000 complete [ 6923.232895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6923.239164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6923.251173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6926.820511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6926.821404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6928.550898] Lustre: server umount lustre-MDT0000 complete [ 6931.946038] LustreError: 214222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6931.962204] LustreError: 214222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6932.578927] LustreError: 214955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787439118 with bad export cookie 5783803099753997522 [ 6932.586498] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6932.594989] LustreError: 214955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6932.894243] Lustre: server umount lustre-MDT0001 complete [ 6942.968151] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6946.596375] Key type lgssc unregistered [ 6947.080709] LNet: 217906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6947.090890] LNetError: 217906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6947.127431] LNet: Removed LNI 192.168.202.137@tcp [ 6948.670246] Key type .llcrypt unregistered [ 6948.671818] Key type ._llcrypt unregistered [ 6972.988613] Key type ._llcrypt registered [ 6972.990343] Key type .llcrypt registered [ 6973.151505] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6974.129226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6974.297429] alg: No test for adler32 (adler32-zlib) [ 6975.406619] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 6975.731580] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6977.503338] Key type lgssc registered [ 6978.914735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6987.170736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6987.200702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6988.741717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6993.828478] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6999.017550] LustreError: 219174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6999.033258] LustreError: 219174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7001.051442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7001.316868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7005.072600] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7011.257098] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7016.132693] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7022.235716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7022.505195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7028.846492] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7033.790486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7036.039909] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7044.286129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7044.337859] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7044.497309] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7049.478971] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7055.545497] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7057.215951] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7069.259718] Lustre: Permanently deactivating lustre-OST0001 [ 7069.275929] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7076.387014] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7087.916457] Lustre: Permanently reactivating lustre-OST0001 [ 7087.943708] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7087.958985] Lustre: Skipped 2 previous similar messages [ 7098.400412] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.417107] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7102.982208] Lustre: server umount lustre-OST0001 complete [ 7103.457553] LustreError: 222394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7108.579479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7108.580915] LustreError: 220994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7108.581644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7108.582565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7108.586249] Lustre: Skipped 1 previous similar message [ 7111.915539] Lustre: server umount lustre-OST0000 complete [ 7118.816151] LustreError: 223079:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 7118.817335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7118.820137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7118.820146] Lustre: Skipped 1 previous similar message [ 7118.835964] Lustre: Skipped 1 previous similar message [ 7123.946028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7123.950222] Lustre: Skipped 2 previous similar messages [ 7125.051350] Lustre: server umount lustre-MDT0000 complete [ 7129.006364] LustreError: 222010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787439314 with bad export cookie 14152499473891470909 [ 7129.015849] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7129.576901] Lustre: server umount lustre-MDT0001 complete [ 7149.371608] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 7152.237418] Key type lgssc unregistered [ 7152.565848] LNet: 224256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7152.571445] LNetError: 224256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7153.651719] LNet: Removed LNI 192.168.202.137@tcp [ 7154.692214] Key type .llcrypt unregistered [ 7154.695899] Key type ._llcrypt unregistered [ 7197.925922] Key type ._llcrypt registered [ 7197.929410] Key type .llcrypt registered [ 7198.092771] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7200.088563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7200.130592] alg: No test for adler32 (adler32-zlib) [ 7201.262788] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 7201.476689] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 7203.215197] Key type lgssc registered [ 7204.718728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7214.207683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7214.232984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7215.611485] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7215.639501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7215.815670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7220.634494] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7227.537162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7227.597147] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7227.616430] Lustre: 226461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7227.833358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7233.045585] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7245.865918] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7254.096158] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7263.569246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7264.093519] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7264.421228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7270.634308] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7270.993264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7278.946273] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7282.667730] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7285.490184] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7291.916751] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7299.909675] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7312.800721] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7312.811831] Lustre: Skipped 1 previous similar message [ 7330.606781] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7330.617404] Lustre: Skipped 1 previous similar message [ 7363.074485] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7363.078878] Lustre: Skipped 4 previous similar messages [ 7435.457605] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7435.465125] Lustre: Skipped 8 previous similar messages [ 7444.977195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7444.978978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7444.992669] LustreError: Skipped 1 previous similar message [ 7445.004401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7445.014857] Lustre: Skipped 1 previous similar message [ 7445.056409] Lustre: Skipped 1 previous similar message [ 7447.463924] Lustre: server umount lustre-OST0000 complete [ 7453.152237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7453.165990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7453.202826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7455.200422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7455.205320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7455.244144] Lustre: Skipped 1 previous similar message [ 7458.773551] Lustre: server umount lustre-MDT0000 complete [ 7463.242702] LustreError: 227204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787439649 with bad export cookie 6386476579304644393 [ 7463.260678] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7463.268322] LustreError: 227204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7463.645855] Lustre: server umount lustre-MDT0001 complete [ 7476.846709] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 7479.941021] Key type lgssc unregistered [ 7480.224835] LNet: 231262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7480.231831] LNetError: 231262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7480.262648] LNet: Removed LNI 192.168.202.137@tcp [ 7481.295736] Key type .llcrypt unregistered [ 7481.298159] Key type ._llcrypt unregistered [ 7511.052854] Key type ._llcrypt registered [ 7511.055782] Key type .llcrypt registered [ 7511.253356] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7513.356563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7513.506960] alg: No test for adler32 (adler32-zlib) [ 7514.585571] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 7514.814882] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 7516.583290] Key type lgssc registered [ 7517.513321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7525.521195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7525.535636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7527.035675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7532.541372] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7536.613723] LustreError: 232678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7536.632754] LustreError: 232678:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7539.979213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7540.369493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7544.566684] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7551.028967] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7556.446642] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7562.616174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7562.981605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7569.853502] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7571.280994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7578.046190] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7597.035684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7597.035776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7597.043288] LustreError: Skipped 1 previous similar message [ 7597.044426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7597.066599] Lustre: Skipped 1 previous similar message [ 7598.117766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7598.131759] Lustre: Skipped 2 previous similar messages [ 7598.595220] Lustre: server umount lustre-OST0000 complete [ 7603.184623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7603.189959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7603.196221] Lustre: Skipped 1 previous similar message [ 7603.202168] Lustre: Skipped 1 previous similar message [ 7609.203321] Lustre: server umount lustre-MDT0000 complete [ 7609.313554] LustreError: 233406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7613.181357] LustreError: 232657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787439799 with bad export cookie 3337187632607611631 [ 7613.185262] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7613.191050] LustreError: 232657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7613.826627] Lustre: server umount lustre-MDT0001 complete [ 7628.409762] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 7631.267735] Key type lgssc unregistered [ 7631.565883] LNet: 236376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7631.572481] LNetError: 236376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7631.585947] LNet: Removed LNI 192.168.202.137@tcp [ 7632.369792] Key type .llcrypt unregistered [ 7632.375359] Key type ._llcrypt unregistered [ 7654.792058] Key type ._llcrypt registered [ 7654.793532] Key type .llcrypt registered [ 7654.893165] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7656.556737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7656.630188] alg: No test for adler32 (adler32-zlib) [ 7657.833295] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 7658.210123] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 7660.055221] Key type lgssc registered [ 7661.770685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7936.941733] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 7940.237644] Key type lgssc unregistered [ 7940.609705] LNet: 238403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7940.623509] LNetError: 238403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7940.654026] LNet: Removed LNI 192.168.202.137@tcp [ 7941.581668] Key type .llcrypt unregistered [ 7941.584627] Key type ._llcrypt unregistered [ 7972.065235] Key type ._llcrypt registered [ 7972.069246] Key type .llcrypt registered [ 7972.191184] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 7990.621605] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7991.613974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7991.643777] alg: No test for adler32 (adler32-zlib) [ 7993.010509] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 7993.373810] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 7995.087276] Key type lgssc registered [ 7996.682582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8033.817950] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7673 sec ========== 19:10:18 (1787440218) [ 8036.035423] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:10:20 (1787440220) === [ 8040.188386] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:10:25 (1787440225) === [ 8088.504637] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 8091.327471] Key type lgssc unregistered [ 8091.561666] LNet: 243649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8091.572596] LNetError: 243649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8091.597429] LNet: Removed LNI 192.168.202.137@tcp [ 8092.292657] Key type .llcrypt unregistered [ 8092.294647] Key type ._llcrypt unregistered