[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 591297538 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002384] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006953] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009128] LSM: Security Framework initializing [ 0.010054] Yama: becoming mindful. [ 0.011034] SELinux: Initializing. [ 0.012058] *** VALIDATE selinux *** [ 0.020391] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025409] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027094] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] *** VALIDATE tmpfs *** [ 0.030468] *** VALIDATE proc *** [ 0.032033] *** VALIDATE cgroup *** [ 0.032959] *** VALIDATE cgroup2 *** [ 0.034217] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041488] debug: unmapping init [mem 0xffffffff84059000-0xffffffff84060fff] [ 0.043931] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044608] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046009] ... bit width: 48 [ 0.047008] ... generic registers: 4 [ 0.048009] ... value mask: 0000ffffffffffff [ 0.049011] ... max period: 00007fffffffffff [ 0.050012] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052310] rcu: Hierarchical SRCU implementation. [ 0.054429] smp: Bringing up secondary CPUs ... [ 0.055531] x86: Booting SMP configuration: [ 0.056024] .... node #0, CPUs: #1 #2 #3 [ 0.059171] smp: Brought up 1 node, 4 CPUs [ 0.061011] smpboot: Max logical packages: 1 [ 0.062017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150469] node 0 deferred pages initialised in 87ms [ 0.155067] devtmpfs: initialized [ 0.156270] x86/mm: Memory block size: 128MB [ 0.159805] gcov: version magic: 0x41383552 [ 0.162171] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165102] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.167318] pinctrl core: initialized pinctrl subsystem [ 0.169168] [ 0.169684] ************************************************************* [ 0.172023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174007] ** ** [ 0.176012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.178014] ** ** [ 0.181011] ** This means that this kernel is built to expose internal ** [ 0.183010] ** IOMMU data structures, which may compromise security on ** [ 0.185014] ** your system. ** [ 0.188011] ** ** [ 0.190010] ** If you see this message and you are not debugging the ** [ 0.192021] ** kernel, report this immediately to your vendor! ** [ 0.195010] ** ** [ 0.198013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.200236] ************************************************************* [ 0.203654] NET: Registered protocol family 16 [ 0.205413] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.208061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.211060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.215071] cpuidle: using governor menu [ 0.218943] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.222496] PCI: Using configuration type 1 for base access [ 0.224126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.236146] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.237051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.239137] cryptd: max_cpu_qlen set to 1000 [ 0.242262] ACPI: Added _OSI(Module Device) [ 0.243022] ACPI: Added _OSI(Processor Device) [ 0.244026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.245013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.250030] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.259314] ACPI: Interpreter enabled [ 0.260058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.263016] ACPI: Using IOAPIC for interrupt routing [ 0.266120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.270422] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.280399] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.283043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.285019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.290082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.295277] acpiphp: Slot [2] registered [ 0.296098] acpiphp: Slot [5] registered [ 0.298194] acpiphp: Slot [6] registered [ 0.300100] acpiphp: Slot [7] registered [ 0.301125] acpiphp: Slot [8] registered [ 0.303111] acpiphp: Slot [9] registered [ 0.304138] acpiphp: Slot [10] registered [ 0.306121] acpiphp: Slot [3] registered [ 0.308100] acpiphp: Slot [4] registered [ 0.309088] acpiphp: Slot [11] registered [ 0.310068] acpiphp: Slot [12] registered [ 0.311046] acpiphp: Slot [13] registered [ 0.312030] acpiphp: Slot [14] registered [ 0.314141] acpiphp: Slot [15] registered [ 0.315091] acpiphp: Slot [16] registered [ 0.317097] acpiphp: Slot [17] registered [ 0.318094] acpiphp: Slot [18] registered [ 0.320085] acpiphp: Slot [19] registered [ 0.321106] acpiphp: Slot [20] registered [ 0.322081] acpiphp: Slot [21] registered [ 0.323079] acpiphp: Slot [22] registered [ 0.325096] acpiphp: Slot [23] registered [ 0.326104] acpiphp: Slot [24] registered [ 0.327094] acpiphp: Slot [25] registered [ 0.330143] acpiphp: Slot [26] registered [ 0.331136] acpiphp: Slot [27] registered [ 0.333093] acpiphp: Slot [28] registered [ 0.334090] acpiphp: Slot [29] registered [ 0.336083] acpiphp: Slot [30] registered [ 0.337075] acpiphp: Slot [31] registered [ 0.338097] PCI host bridge to bus 0000:00 [ 0.339015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.341019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.344013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.347021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.349019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.352028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.354171] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.356994] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.361000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.370800] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.376566] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.378026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.380014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.382014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.384588] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.388890] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.391041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.394826] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.399018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.412018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.418744] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.424749] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.468017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.487019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.604027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.647178] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.663016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.679026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.727017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.748486] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.785023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.829019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.875017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.896000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.911015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.926022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.972018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.000182] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.019017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.041016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.088019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.111430] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.133020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.172020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.261020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.281025] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.284345] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.286309] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.288377] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.290211] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.296032] iommu: Default domain type: Passthrough [ 1.297400] SCSI subsystem initialized [ 1.299125] ACPI: bus type USB registered [ 1.300083] usbcore: registered new interface driver usbfs [ 1.302063] usbcore: registered new interface driver hub [ 1.303057] usbcore: registered new device driver usb [ 1.304102] pps_core: LinuxPPS API ver. 1 registered [ 1.305006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.307046] PTP clock support registered [ 1.309053] EDAC MC: Ver: 3.0.0 [ 1.311111] PCI: Using ACPI for IRQ routing [ 1.311833] NetLabel: Initializing [ 1.313006] NetLabel: domain hash size = 128 [ 1.313954] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.315052] NetLabel: unlabeled traffic allowed by default [ 1.316109] vgaarb: loaded [ 1.317239] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.319010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.322339] clocksource: Switched to clocksource kvm-clock [ 1.425657] VFS: Disk quotas dquot_6.6.0 [ 1.427527] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.429907] *** VALIDATE ramfs *** [ 1.431062] *** VALIDATE hugetlbfs *** [ 1.432460] pnp: PnP ACPI init [ 1.434597] pnp: PnP ACPI: found 6 devices [ 1.449835] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.452798] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.455073] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.457243] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.459375] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.461724] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.464374] NET: Registered protocol family 2 [ 1.466864] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.471882] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.475066] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.480349] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.483705] TCP: Hash tables configured (established 65536 bind 65536) [ 1.486534] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.489565] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.492637] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.495344] NET: Registered protocol family 1 [ 1.498528] RPC: Registered named UNIX socket transport module. [ 1.500504] RPC: Registered udp transport module. [ 1.501978] RPC: Registered tcp transport module. [ 1.503021] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.504945] NET: Registered protocol family 44 [ 1.506603] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.508776] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.510810] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.513034] PCI: CLS 0 bytes, default 64 [ 1.514565] Unpacking initramfs... [ 2.905725] debug: unmapping init [mem 0xffff9715fcc54000-0xffff9715fffbffff] [ 2.909822] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.912717] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.915558] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.413737] Initialise system trusted keyrings [ 3.415086] Key type blacklist registered [ 3.416507] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.424782] zbud: loaded [ 3.427960] *** VALIDATE nfs *** [ 3.429032] *** VALIDATE nfs4 *** [ 3.430461] pstore: using deflate compression [ 3.433662] Platform Keyring initialized [ 3.528047] NET: Registered protocol family 38 [ 3.530261] Key type asymmetric registered [ 3.532063] Asymmetric key parser 'x509' registered [ 3.534188] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.538125] io scheduler mq-deadline registered [ 3.540096] io scheduler kyber registered [ 3.541879] io scheduler bfq registered [ 3.545181] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.548149] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.550262] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.553090] ACPI: Power Button [PWRF] [ 3.557671] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.563948] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.594518] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.620936] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.682036] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.709351] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.736048] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.740704] Non-volatile memory driver v1.3 [ 3.741814] Linux agpgart interface v0.103 [ 3.770564] virtio_blk virtio1: [vda] 145872 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.772881] vda: detected capacity change from 0 to 74686464 [ 3.824764] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.826590] vdb: detected capacity change from 0 to 1073741824 [ 3.854981] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.859135] vdc: detected capacity change from 0 to 2621440000 [ 3.889160] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.891765] vdd: detected capacity change from 0 to 2621440000 [ 3.916551] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.919429] vde: detected capacity change from 0 to 4294967296 [ 3.946767] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.948839] vdf: detected capacity change from 0 to 4294967296 [ 3.973887] libphy: Fixed MDIO Bus: probed [ 3.987151] usbcore: registered new interface driver usbserial_generic [ 3.989841] usbserial: USB Serial support registered for generic [ 3.992120] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.995577] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.996944] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.998984] mousedev: PS/2 mouse device common for all mice [ 4.002038] rtc_cmos 00:05: RTC can wake from S4 [ 4.004513] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.008251] rtc_cmos 00:05: registered as rtc0 [ 4.010341] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.011465] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.014049] intel_pstate: CPU model not supported [ 4.019050] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.022982] hid: raw HID events driver (C) Jiri Kosina [ 4.024765] usbcore: registered new interface driver usbhid [ 4.026174] usbhid: USB HID core driver [ 4.027419] drop_monitor: Initializing network drop monitor service [ 4.029149] Initializing XFRM netlink socket [ 4.030719] NET: Registered protocol family 10 [ 4.033297] Segment Routing with IPv6 [ 4.034687] NET: Registered protocol family 17 [ 4.036379] mpls_gso: MPLS GSO support [ 4.041060] RAS: Correctable Errors collector initialized. [ 4.042957] AVX version of gcm_enc/dec engaged. [ 4.044303] AES CTR mode by8 optimization enabled [ 4.116332] sched_clock: Marking stable (4116312804, 0)->(4995057791, -878744987) [ 4.119712] registered taskstats version 1 [ 4.121572] Loading compiled-in X.509 certificates [ 4.123041] zswap: loaded using pool lzo/zbud [ 4.143908] Key type big_key registered [ 4.163298] Key type encrypted registered [ 4.166269] ima: No TPM chip found, activating TPM-bypass! [ 4.168515] ima: Allocated hash algorithm: sha1 [ 4.170220] ima: No architecture policies found [ 4.172112] evm: Initialising EVM extended attributes: [ 4.174141] evm: security.selinux [ 4.175347] evm: security.ima [ 4.176497] evm: security.capability [ 4.177817] evm: HMAC attrs: 0x1 [ 4.180307] rtc_cmos 00:05: setting system clock to 2026-07-30 22:40:07 UTC (1785451207) [ 4.186721] debug: unmapping init [mem 0xffffffff85003000-0xffffffff851fffff] [ 4.189753] debug: unmapping init [mem 0xffffffff83d82000-0xffffffff84058fff] [ 4.198113] Write protecting the kernel read-only data: 28672k [ 4.201528] debug: unmapping init [mem 0xffffffff82403000-0xffffffff825fffff] [ 4.204535] debug: unmapping init [mem 0xffffffff82d14000-0xffffffff82dfffff] [ 4.242197] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.250788] systemd[1]: Detected virtualization kvm. [ 4.252691] systemd[1]: Detected architecture x86-64. [ 4.254193] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.281194] systemd[1]: No hostname configured. [ 4.282849] systemd[1]: Set hostname to . [ 4.284668] random: systemd: uninitialized urandom read (16 bytes read) [ 4.286521] systemd[1]: Initializing machine ID from random generator. [ 4.336772] random: ln: uninitialized urandom read (6 bytes read) [ 4.427984] random: systemd: uninitialized urandom read (16 bytes read) [ 4.430744] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.434938] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.438161] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.043554] device-mapper: uevent: version 1.0.3 [ 5.045564] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK [ 5.276931] random: fast init done ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.820473] virtio_net virtio0 ens2: renamed from eth0 [ 5.853279] scsi host0: ata_piix [ 5.861302] scsi host1: ata_piix [ 5.863259] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.865822] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.445024] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.684691] random: crng init done [ 10.686293] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.144865] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.242215] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.527191] SELinux: Disabled at runtime. [ 12.597630] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.606440] systemd[1]: Detected virtualization kvm. [ 12.608917] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.153468] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.156882] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.162092] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.166352] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.169116] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.175748] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.195307] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ 13.258827] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.578542] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.860600] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.928125] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.053785] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.066238] EDAC sbridge: Ver: 1.1.2 [ 15.593929] Key type dns_resolver registered [ 15.901468] NFS: Registering the id_resolver key type [ 15.903474] Key type id_resolver registered [ 15.905039] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg122-server login: [ 33.258107] hrtimer: interrupt took 1966202 ns [ 73.225535] libcfs: loading out-of-tree module taints kernel. [ 73.330967] Key type ._llcrypt registered [ 73.334902] Key type .llcrypt registered [ 73.522180] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 106.182507] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 109.565089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 109.588994] alg: No test for adler32 (adler32-zlib) [ 111.797122] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 113.348367] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 115.192559] Key type lgssc registered [ 118.927310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 147.840471] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 205.565269] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 225.450388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 225.538889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.755215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 226.816160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 226.974933] Lustre: lustre-MDT0000: new disk, initializing [ 227.048930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 227.080430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 233.449235] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.202896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 256.440771] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 256.571893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 256.578349] Lustre: Skipped 1 previous similar message [ 256.873863] Lustre: lustre-MDT0001: new disk, initializing [ 257.085138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.162911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 257.196506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 264.773299] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.611075] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 284.219535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 284.494513] Lustre: lustre-OST0000: new disk, initializing [ 284.498919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 284.510483] Lustre: 8462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 284.593357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 292.278301] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 292.394549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 292.414727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 292.493462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 312.166840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 312.376887] Lustre: lustre-OST0001: new disk, initializing [ 312.382651] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 312.391800] Lustre: 9537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 312.521124] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 320.075053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 320.081288] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 320.157118] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 321.157254] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 337.989155] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 346.700659] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 357.404340] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing check_logdir /tmp/testlogs/ [ 365.136594] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing yml_node [ 372.989656] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 376.350778] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 379.247281] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 381.392132] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 30 18:46:23 EDT 2026 [ 403.460272] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 405.699369] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 425.952539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 425.966392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.991430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 428.002593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 428.006821] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.045903] Lustre: Skipped 2 previous similar messages [ 432.591781] Lustre: server umount lustre-MDT0000 complete [ 437.212711] LustreError: 7938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785451640 with bad export cookie 9445539413741201444 [ 437.219937] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 437.233528] LustreError: 7938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 438.240612] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.242301] LustreError: 11967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.271834] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 438.271842] Lustre: Skipped 3 previous similar messages [ 438.310283] LustreError: 11967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 442.274038] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 442.275850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 442.291561] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 442.315516] Lustre: Skipped 2 previous similar messages [ 443.822791] Lustre: server umount lustre-MDT0001 complete [ 450.385050] Lustre: server umount lustre-OST0000 complete [ 455.310993] Lustre: server umount lustre-OST0001 complete [ 465.162939] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 477.426950] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 529.718455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.021790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 530.047229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 530.142616] Lustre: lustre-MDT0000: new disk, initializing [ 530.224665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 530.243449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 535.869563] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 548.641449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.870716] Lustre: 15100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 548.956765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 548.964430] Lustre: Skipped 1 previous similar message [ 549.073464] Lustre: lustre-MDT0001: new disk, initializing [ 549.191399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 549.252222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 549.272421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 555.081443] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 567.534667] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 574.692024] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 582.573559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 582.796627] Lustre: lustre-OST0000: new disk, initializing [ 582.799561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 582.803987] Lustre: 17068:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 582.906064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 584.610927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 584.628124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 584.753648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 590.232240] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 602.279726] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 607.742527] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 608.016620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 614.502405] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 614.771888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 620.512385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 620.526747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.532238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 620.542133] Lustre: Skipped 2 previous similar messages [ 620.558471] Lustre: Skipped 1 previous similar message [ 623.413100] Lustre: server umount lustre-OST0000 complete [ 630.757785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.767762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 630.773620] Lustre: Skipped 1 previous similar message [ 630.790730] Lustre: Skipped 1 previous similar message [ 634.609087] Lustre: server umount lustre-MDT0000 complete [ 635.880309] LustreError: 15110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.909688] LustreError: 15110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 639.696890] LustreError: 15092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785451843 with bad export cookie 9445539413741202970 [ 639.703407] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.713141] LustreError: 15092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 640.049878] Lustre: server umount lustre-MDT0001 complete [ 644.766889] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 18:50:46 (1785451846) [ 654.137923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.965460] LustreError: 19336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.090668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 660.450950] LustreError: 19337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 660.461844] LustreError: 19337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 661.699513] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 670.692124] LustreError: 19337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.705246] LustreError: 19337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 672.128613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 679.164686] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 690.492324] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 697.479888] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 706.831034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 713.830108] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 724.807404] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 735.212171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 735.225684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 735.236354] LustreError: Skipped 1 previous similar message [ 735.237316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 735.250656] Lustre: Skipped 1 previous similar message [ 737.114658] Lustre: server umount lustre-OST0000 complete [ 744.425543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 747.913872] Lustre: server umount lustre-MDT0000 complete [ 750.565824] LustreError: 19337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 750.600168] LustreError: 19337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 753.408351] LustreError: 21183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785451956 with bad export cookie 9445539413741203852 [ 753.410832] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 753.425067] LustreError: 21183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 754.227472] Lustre: server umount lustre-MDT0001 complete [ 772.382671] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 777.773326] Key type lgssc unregistered [ 778.293790] LNet: 22876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.304080] LNetError: 22876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.368729] LNet: Removed LNI 192.168.201.122@tcp [ 781.100375] Key type .llcrypt unregistered [ 781.101808] Key type ._llcrypt unregistered [ 813.005536] Key type ._llcrypt registered [ 813.008229] Key type .llcrypt registered [ 813.116170] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 814.412460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 814.628132] alg: No test for adler32 (adler32-zlib) [ 815.705913] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 815.941276] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 817.695892] Key type lgssc registered [ 819.501741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 828.855964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 828.870202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 830.723472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 837.062230] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 840.181111] LustreError: 24291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 840.206107] LustreError: 24291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 845.281514] LustreError: 24292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 848.392946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 848.722655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 854.857704] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 863.745103] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 871.516817] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 879.924282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 880.339711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 887.296081] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 898.057927] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 899.821589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 919.520527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 919.523282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 919.530986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 920.547968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 920.550832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 920.575844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.644269] Lustre: Skipped 1 previous similar message [ 925.716213] LustreError: 26113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.220734] Lustre: server umount lustre-OST0000 complete [ 935.904516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.909052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 935.922989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 940.034517] Lustre: server umount lustre-MDT0000 complete [ 941.031773] LustreError: 24292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 941.057906] LustreError: 24292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 945.143683] LustreError: 24273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785452148 with bad export cookie 11386016452355334998 [ 945.159338] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 945.164849] LustreError: 24273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 945.678479] Lustre: server umount lustre-MDT0001 complete [ 960.758195] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 964.504428] Key type lgssc unregistered [ 964.867592] LNet: 28272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.878675] LNetError: 28272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.903136] LNet: Removed LNI 192.168.201.122@tcp [ 965.987224] Key type .llcrypt unregistered [ 965.992585] Key type ._llcrypt unregistered [ 999.850842] Key type ._llcrypt registered [ 999.852659] Key type .llcrypt registered [ 999.959646] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 1001.267308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1001.349610] alg: No test for adler32 (adler32-zlib) [ 1002.629575] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 1002.924411] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 1004.671176] Key type lgssc registered [ 1006.357986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1019.407815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1019.526428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.595384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.070445] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1031.151514] LustreError: 29688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1031.195710] LustreError: 29688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1036.256299] LustreError: 29689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.026349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1040.644312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1046.598841] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1058.400080] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1065.321162] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1082.944456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1083.168873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1091.390713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1092.635872] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1102.533763] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1111.010061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1111.019079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.045040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1112.037228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1112.050270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1112.051189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.091760] Lustre: Skipped 1 previous similar message [ 1116.790150] Lustre: server umount lustre-OST0000 complete [ 1122.274413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.280389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1122.292380] Lustre: Skipped 1 previous similar message [ 1122.304120] Lustre: Skipped 1 previous similar message [ 1127.396352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1127.411862] Lustre: Skipped 1 previous similar message [ 1127.749861] Lustre: server umount lustre-MDT0000 complete [ 1132.311774] LustreError: 29668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785452335 with bad export cookie 5356294553763588528 [ 1132.324485] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1132.327749] LustreError: 29668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1133.032201] Lustre: server umount lustre-MDT0001 complete [ 1149.270888] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 1154.098197] Key type lgssc unregistered [ 1154.479569] LNet: 33571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.487621] LNetError: 33571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1154.519921] LNet: Removed LNI 192.168.201.122@tcp [ 1155.507174] Key type .llcrypt unregistered [ 1155.508844] Key type ._llcrypt unregistered [ 1185.890817] Key type ._llcrypt registered [ 1185.894876] Key type .llcrypt registered [ 1186.014741] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 1187.423262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1187.641278] alg: No test for adler32 (adler32-zlib) [ 1188.853736] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 1189.224179] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 1190.935193] Key type lgssc registered [ 1192.610546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1202.040037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1202.071258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.581941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.898300] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1212.897110] LustreError: 34986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.906946] LustreError: 34986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1218.029338] LustreError: 34987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1220.109544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.639573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1226.083455] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1234.687435] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1241.523656] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1249.147072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1249.815473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1261.002308] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1273.230338] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1273.481025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1285.607371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1285.615106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.628182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1289.196701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1289.199637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1289.210859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.363449] Lustre: server umount lustre-OST0000 complete [ 1297.381852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1297.393547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.417759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1297.423691] Lustre: Skipped 1 previous similar message [ 1299.425207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1299.432733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.914882] Lustre: server umount lustre-MDT0000 complete [ 1304.561959] LustreError: 34981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.584222] LustreError: 34981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1305.599679] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785452508 with bad export cookie 6640093254689895294 [ 1305.600445] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1305.611846] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1306.141340] Lustre: server umount lustre-MDT0001 complete [ 1320.962540] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 1324.393709] Key type lgssc unregistered [ 1324.723229] LNet: 38534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.735597] LNetError: 38534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.756722] LNet: Removed LNI 192.168.201.122@tcp [ 1326.160064] Key type .llcrypt unregistered [ 1326.164147] Key type ._llcrypt unregistered [ 1356.643803] Key type ._llcrypt registered [ 1356.647120] Key type .llcrypt registered [ 1356.858076] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 1358.243163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1358.393165] alg: No test for adler32 (adler32-zlib) [ 1359.410905] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 1359.720885] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 1361.511274] Key type lgssc registered [ 1362.623161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1369.711926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1369.736944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1371.233831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.967166] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1380.841785] LustreError: 39957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1380.872233] LustreError: 39957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1385.151626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1385.823088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1390.260725] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1398.916641] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1405.249267] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1412.979158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1413.273373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1416.390080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1422.258942] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1431.902844] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1439.137982] Lustre: lustre-OST0000: Not available for connect from 192.168.201.22@tcp (stopping) [ 1441.248418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1441.251093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1441.267161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1441.280447] Lustre: Skipped 1 previous similar message [ 1441.309524] Lustre: Skipped 1 previous similar message [ 1444.210040] Lustre: lustre-OST0000: Not available for connect from 192.168.201.22@tcp (stopping) [ 1445.048320] Lustre: server umount lustre-OST0000 complete [ 1470.946323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1470.950078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1470.967885] Lustre: Skipped 1 previous similar message [ 1470.986114] Lustre: Skipped 1 previous similar message [ 1476.065121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1476.076289] Lustre: Skipped 1 previous similar message [ 1476.419995] Lustre: server umount lustre-MDT0000 complete [ 1480.734522] LustreError: 39939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785452684 with bad export cookie 11050276871794614232 [ 1480.749471] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1487.338673] Lustre: server umount lustre-MDT0001 complete [ 1500.368530] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 1503.912829] Key type lgssc unregistered [ 1504.085676] LNet: 43666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.088795] LNetError: 43666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1504.099380] LNet: Removed LNI 192.168.201.122@tcp [ 1504.888130] Key type .llcrypt unregistered [ 1504.889975] Key type ._llcrypt unregistered [ 1538.338866] Key type ._llcrypt registered [ 1538.341465] Key type .llcrypt registered [ 1538.499658] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 1539.190829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1539.213437] alg: No test for adler32 (adler32-zlib) [ 1540.280836] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 1540.552890] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 1542.433107] Key type lgssc registered [ 1543.687543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1552.716490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1552.727422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1554.109174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1559.420116] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1564.646373] LustreError: 45081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.672983] LustreError: 45081:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1568.895992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.477851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1575.749728] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1585.382424] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1593.186888] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1601.076390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1601.443171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.856350] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1617.994659] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1624.047735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1635.809307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1635.820073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1635.845492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1639.366251] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.22@tcp (stopping) [ 1639.394611] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.423606] Lustre: Skipped 1 previous similar message [ 1641.555958] Lustre: server umount lustre-MDT0000 complete [ 1644.400832] LustreError: 46476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1645.623147] LustreError: 45063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785452848 with bad export cookie 260471079525924178 [ 1645.630369] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1649.519184] LustreError: 46476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1649.559338] LustreError: 46476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1649.606973] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.22@tcp (stopping) [ 1649.615116] Lustre: Skipped 3 previous similar messages [ 1649.642383] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.018066] Lustre: server umount lustre-MDT0001 complete [ 1674.127486] Lustre: server umount lustre-OST0000 complete [ 1698.414612] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 1703.337477] Key type lgssc unregistered [ 1703.933639] LNet: 48969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.948300] LNetError: 48969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.985786] LNet: Removed LNI 192.168.201.122@tcp [ 1705.360365] Key type .llcrypt unregistered [ 1705.361674] Key type ._llcrypt unregistered [ 1737.216556] Key type ._llcrypt registered [ 1737.218306] Key type .llcrypt registered [ 1737.321374] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 1738.541046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1738.665632] alg: No test for adler32 (adler32-zlib) [ 1740.203358] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 1740.526307] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 1742.255494] Key type lgssc registered [ 1743.852855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1753.816909] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1753.841572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1769.442150] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1774.560590] LustreError: 50359:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 1801.312090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.417145] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 1819.437309] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1985.238398] Lustre: server umount lustre-OST0000 complete [ 2009.413137] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2013.021762] Key type lgssc unregistered [ 2013.433326] LNet: 52046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.449857] LNetError: 52046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.508720] LNet: Removed LNI 192.168.201.122@tcp [ 2015.669292] Key type .llcrypt unregistered [ 2015.679072] Key type ._llcrypt unregistered [ 2047.622961] Key type ._llcrypt registered [ 2047.624720] Key type .llcrypt registered [ 2047.862066] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2050.207408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2050.495027] alg: No test for adler32 (adler32-zlib) [ 2051.867792] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 2052.232113] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 2053.946158] Key type lgssc registered [ 2055.835096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2066.281945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2066.294677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.075055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2073.539578] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2078.689929] LustreError: 53464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2078.713903] LustreError: 53464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2083.814256] LustreError: 53465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.951896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.560890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2089.925542] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2098.535134] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2104.925974] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2113.692421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2114.098547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.134535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2122.999920] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2132.779798] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2135.523341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2135.526648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.544912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2139.620994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2139.627691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2139.655112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.694052] Lustre: server umount lustre-OST0000 complete [ 2149.858583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.861410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.874790] Lustre: Skipped 1 previous similar message [ 2149.894419] Lustre: Skipped 2 previous similar messages [ 2154.980911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2154.989142] Lustre: Skipped 1 previous similar message [ 2160.099356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2161.119133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2161.710720] Lustre: server umount lustre-MDT0000 complete [ 2165.218769] LustreError: 53464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.242809] LustreError: 53464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2166.906647] LustreError: 53446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785453370 with bad export cookie 1317106864365119667 [ 2166.917643] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2166.931505] LustreError: 53446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2167.427226] Lustre: server umount lustre-MDT0001 complete [ 2180.450194] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2183.918941] Key type lgssc unregistered [ 2184.193084] LNet: 56998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2184.199270] LNetError: 56998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2184.216558] LNet: Removed LNI 192.168.201.122@tcp [ 2185.439136] Key type .llcrypt unregistered [ 2185.450683] Key type ._llcrypt unregistered [ 2216.014836] Key type ._llcrypt registered [ 2216.019351] Key type .llcrypt registered [ 2216.131905] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2217.809214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2218.190611] alg: No test for adler32 (adler32-zlib) [ 2219.393851] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 2219.604354] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 2221.239160] Key type lgssc registered [ 2222.318544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2231.550971] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2231.575436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2247.200293] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2252.320375] LustreError: 58387:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 2279.135846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.972716] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2300.030861] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2308.079846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.821406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.925215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2313.962540] LustreError: 59426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.988533] LustreError: 59426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2314.079836] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2319.364396] LustreError: 59424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.395216] LustreError: 59424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2324.467550] LustreError: 59426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.533904] LustreError: 59426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2327.487790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.197223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2334.253273] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2343.618473] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2350.788580] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2353.633117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2353.646085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2353.674804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2354.149170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.267487] LustreError: 58400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.496881] Lustre: server umount lustre-OST0000 complete [ 2384.352901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2384.358678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2384.372358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2384.383562] Lustre: Skipped 2 previous similar messages [ 2388.967367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2388.977849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2390.943789] Lustre: server umount lustre-MDT0000 complete [ 2395.106255] LustreError: 59430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.134227] LustreError: 59430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2395.395793] LustreError: 58394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785453598 with bad export cookie 8466312124921834504 [ 2395.411953] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2395.415073] LustreError: 58394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2401.863553] Lustre: server umount lustre-MDT0001 complete [ 2415.685799] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2418.703921] Key type lgssc unregistered [ 2419.041049] LNet: 62132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.050431] LNetError: 62132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.068174] LNet: Removed LNI 192.168.201.122@tcp [ 2420.041127] Key type .llcrypt unregistered [ 2420.042713] Key type ._llcrypt unregistered [ 2448.697607] Key type ._llcrypt registered [ 2448.699651] Key type .llcrypt registered [ 2448.773707] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2449.708960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2449.889367] alg: No test for adler32 (adler32-zlib) [ 2451.359392] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 2451.584613] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 2453.271171] Key type lgssc registered [ 2454.715201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2463.721602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2463.773066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2465.519771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2470.931308] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2474.986274] LustreError: 63545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.011447] LustreError: 63545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2480.103117] LustreError: 63546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.333117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.808464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2488.001480] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2498.662461] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2508.099889] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2517.565777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2517.987980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.437541] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2535.425107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2545.292881] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2564.065388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2564.077585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2564.095465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2566.113440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2566.118938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2570.147071] Lustre: server umount lustre-OST0000 complete [ 2574.816607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2574.821294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.846515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2574.850679] Lustre: Skipped 1 previous similar message [ 2580.859355] Lustre: server umount lustre-MDT0000 complete [ 2581.474219] LustreError: 63540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.486444] LustreError: 63540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2585.909898] LustreError: 63526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785453789 with bad export cookie 15126397213108713522 [ 2585.917587] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2585.925034] LustreError: 63526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2586.552565] Lustre: server umount lustre-MDT0001 complete [ 2602.126692] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2605.983849] Key type lgssc unregistered [ 2606.449772] LNet: 67109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2606.466601] LNetError: 67109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2606.491505] LNet: Removed LNI 192.168.201.122@tcp [ 2607.877342] Key type .llcrypt unregistered [ 2607.878935] Key type ._llcrypt unregistered [ 2639.945976] Key type ._llcrypt registered [ 2639.947619] Key type .llcrypt registered [ 2640.195332] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2642.052922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2642.211764] alg: No test for adler32 (adler32-zlib) [ 2643.326273] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 2643.556599] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 2645.223294] Key type lgssc registered [ 2646.950808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2657.362989] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 19:24:19 (1785453859) [ 2665.225977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2665.264244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2666.782088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2672.285555] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2677.222041] LustreError: 68815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.247926] LustreError: 68815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2682.342185] LustreError: 68814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.135430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.642966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2689.147602] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2699.113409] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2707.474311] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2716.959935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2717.304954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.775401] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2736.860474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2740.436366] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2747.800871] Lustre: Failing over lustre-MDT0000 [ 2748.958820] Lustre: server umount lustre-MDT0000 complete [ 2750.438216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2750.440747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2750.467832] LustreError: 69922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2752.480835] LustreError: 68815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2752.485331] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.614027] LustreError: 71200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2757.631451] LustreError: 71200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2763.552026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2763.630551] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2763.722987] Lustre: *** cfs_fail_loc=135, val=0*** [ 2763.788599] LustreError: 71906:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2763.792400] LustreError: 71906:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.122@tcp: cfg command failed: rc = -2 [ 2763.796388] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2763.796388] [ 2763.801162] LustreError: MGC192.168.201.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2763.813867] LustreError: 71894:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2763.820243] LustreError: 71894:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2763.831051] LustreError: 71894:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2763.836159] LustreError: 71894:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2763.854841] Lustre: server umount lustre-MDT0000 complete [ 2763.857092] LustreError: 71894:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2767.842413] LustreError: 68814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.868566] LustreError: 68814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2776.092968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.518728] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2776.548391] LustreError: 72251:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.201.122@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000072197504) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2776.577462] LustreError: 68806:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2777.222510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2777.327356] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2778.552071] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2782.733255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2782.821953] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2782.978203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2787.594076] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2803.236118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2803.242674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2803.297844] Lustre: Skipped 2 previous similar messages [ 2803.335043] Lustre: Skipped 1 previous similar message [ 2808.037870] Lustre: server umount lustre-OST0000 complete [ 2818.532596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.542721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2818.547819] Lustre: Skipped 1 previous similar message [ 2820.673875] Lustre: server umount lustre-MDT0000 complete [ 2823.654835] LustreError: 71200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.681843] LustreError: 71200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2828.658625] LustreError: 68795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785454031 with bad export cookie 16798719827740671914 [ 2828.670390] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2828.691085] LustreError: 68795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2830.304228] Lustre: server umount lustre-MDT0001 complete [ 2853.603315] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 2857.275407] Key type lgssc unregistered [ 2857.696185] LNet: 73721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2857.712184] LNetError: 73721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2858.792977] LNet: Removed LNI 192.168.201.122@tcp [ 2860.094990] Key type .llcrypt unregistered [ 2860.098540] Key type ._llcrypt unregistered [ 2890.643670] Key type ._llcrypt registered [ 2890.645019] Key type .llcrypt registered [ 2890.781857] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 2892.196874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2892.399904] alg: No test for adler32 (adler32-zlib) [ 2893.413542] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 2893.768679] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 2895.567255] Key type lgssc registered [ 2897.127132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2907.650958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2907.714936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.175409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2916.054788] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2919.410736] LustreError: 75134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.435411] LustreError: 75134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2924.515852] LustreError: 75135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2928.961926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.638163] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2930.051705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2936.848423] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2950.422982] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2958.722171] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2970.798030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2971.294487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2977.659869] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 2979.523231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2985.957820] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2993.249956] Lustre: Failing over lustre-MDT0000 [ 2993.560947] LustreError: 77578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2993.673517] Lustre: server umount lustre-MDT0000 complete [ 2995.169671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.196746] Lustre: Skipped 2 previous similar messages [ 2998.639191] LustreError: 75129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2998.658579] LustreError: 75129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3003.759581] LustreError: 75130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3003.787973] LustreError: 75130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3006.648907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3006.820751] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3007.194554] Lustre: *** cfs_fail_loc=a05, val=0*** [ 3007.402545] LustreError: 78223:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 3007.419543] LustreError: 78223:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.122@tcp: cfg command failed: rc = -9 [ 3007.443976] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3007.443976] [ 3007.462510] LustreError: MGC192.168.201.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3007.479426] LustreError: 78211:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 3007.491392] LustreError: 78211:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 3007.506180] LustreError: 78211:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3007.516260] LustreError: 78211:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3007.548429] Lustre: server umount lustre-MDT0000 complete [ 3007.555753] LustreError: 78211:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 3015.018802] LustreError: 75131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3015.044554] LustreError: 75131:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3016.814018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.068218] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3017.088913] LustreError: 75127:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3017.099086] LustreError: 78571:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.201.122@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000004a5e9103) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3017.475823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3017.556839] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3020.157651] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3022.653440] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3022.836422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3022.885949] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3022.959724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 3037.170185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3037.181680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3037.221697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3038.190272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3040.072613] Lustre: server umount lustre-OST0000 complete [ 3048.419162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3048.420713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3048.477425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3048.479467] Lustre: Skipped 2 previous similar messages [ 3051.263108] Lustre: server umount lustre-MDT0000 complete [ 3053.538909] LustreError: 75135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.559386] LustreError: 75135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3055.818537] LustreError: 75117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785454259 with bad export cookie 10493791422251791208 [ 3055.829805] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3055.837256] LustreError: 75117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3056.682826] Lustre: server umount lustre-MDT0001 complete [ 3071.059092] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 3075.154490] Key type lgssc unregistered [ 3075.665612] LNet: 80005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3075.693491] LNetError: 80005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3076.714718] LNet: Removed LNI 192.168.201.122@tcp [ 3077.656195] Key type .llcrypt unregistered [ 3077.660537] Key type ._llcrypt unregistered [ 3095.553277] Key type ._llcrypt registered [ 3095.555549] Key type .llcrypt registered [ 3095.686280] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 3096.232275] Key type .llcrypt unregistered [ 3096.233703] Key type ._llcrypt unregistered [ 3128.889804] Key type ._llcrypt registered [ 3128.896204] Key type .llcrypt registered [ 3129.001656] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 3130.310216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3130.460793] alg: No test for adler32 (adler32-zlib) [ 3131.900934] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 3132.372453] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 3134.215168] Key type lgssc registered [ 3136.364717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3147.453879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3147.498225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.322702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3157.243520] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3159.523284] LustreError: 81909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.560363] LustreError: 81909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3164.655830] LustreError: 81908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.777553] LustreError: 81909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.530357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.921769] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3174.227192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3179.577983] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3192.495674] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3201.861868] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3211.097297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3211.468080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3218.634198] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3219.686362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 3230.478058] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3248.377767] Lustre: server umount lustre-OST0000 complete [ 3255.796924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.816159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3255.823447] Lustre: Skipped 1 previous similar message [ 3255.842372] Lustre: Skipped 1 previous similar message [ 3260.897982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3260.907911] Lustre: Skipped 1 previous similar message [ 3264.115537] Lustre: server umount lustre-MDT0000 complete [ 3266.020318] Lustre: 81221:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785454453/real 1785454453] req@00000000dc878cd8 x1872184564811264/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785454469 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3266.022192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.035493] Lustre: 81221:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3266.047991] LustreError: 81909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3266.064274] LustreError: 81909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3268.876542] LustreError: 81889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785454472 with bad export cookie 7302790151795019627 [ 3268.880666] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3268.886922] LustreError: 81889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3269.289139] Lustre: server umount lustre-MDT0001 complete [ 3285.469331] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 3288.608828] Key type lgssc unregistered [ 3288.906206] LNet: 85468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3288.928214] LNetError: 85468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3288.955657] LNet: Removed LNI 192.168.201.122@tcp [ 3290.427098] Key type .llcrypt unregistered [ 3290.429084] Key type ._llcrypt unregistered [ 3323.051346] Key type ._llcrypt registered [ 3323.053344] Key type .llcrypt registered [ 3323.161759] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 3325.387030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3325.493447] alg: No test for adler32 (adler32-zlib) [ 3326.636448] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 3327.019356] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 3328.799177] Key type lgssc registered [ 3330.738449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3341.206647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3341.219553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.855355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3348.155320] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3353.080795] LustreError: 86882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.131087] LustreError: 86882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3358.184859] LustreError: 86883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.061359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.668854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3366.319424] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3376.809665] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3386.455640] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3397.126739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3398.048121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3406.568787] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3413.457934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3418.296101] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3427.529553] Lustre: server umount lustre-OST0000 complete [ 3433.954188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3433.958685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3433.961214] Lustre: Skipped 1 previous similar message [ 3433.980397] Lustre: Skipped 1 previous similar message [ 3439.077037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3439.096403] Lustre: Skipped 1 previous similar message [ 3443.465766] Lustre: server umount lustre-MDT0000 complete [ 3444.193815] LustreError: 86882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3444.218707] LustreError: 86882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3445.216871] Lustre: 86193:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785454632/real 1785454632] req@000000003b6a75b3 x1872184769281408/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785454648 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3445.258036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3446.308200] Lustre: 86194:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785454633/real 1785454633] req@000000001fc55025 x1872184769281792/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785454649 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3450.149245] Lustre: 86192:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785454637/real 1785454637] req@000000009d05e08f x1872184769283328/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785454653 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3450.377328] LustreError: 86878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3450.410834] LustreError: 86878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3450.472067] LustreError: 86864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785454653 with bad export cookie 4482939352471689436 [ 3450.485343] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3450.488020] LustreError: 86864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3451.679084] Lustre: server umount lustre-MDT0001 complete [ 3469.598411] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 3474.632262] Key type lgssc unregistered [ 3475.033733] LNet: 90420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3475.042529] LNetError: 90420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3475.058970] LNet: Removed LNI 192.168.201.122@tcp [ 3476.490734] Key type .llcrypt unregistered [ 3476.495412] Key type ._llcrypt unregistered [ 3504.449781] Key type ._llcrypt registered [ 3504.450856] Key type .llcrypt registered [ 3504.521170] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 3505.547346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3505.567400] alg: No test for adler32 (adler32-zlib) [ 3507.020008] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 3507.345167] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 3508.983173] Key type lgssc registered [ 3510.926867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3522.403490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3522.435916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.933760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3530.395610] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3534.307940] LustreError: 91837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3534.320718] LustreError: 91837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3539.425503] LustreError: 91838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3541.441608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3542.117918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3549.280949] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3560.414340] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3569.813861] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3580.477398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3581.144993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3590.490156] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3595.468314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3600.904413] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3616.242833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3616.250458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3616.273301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3621.056597] Lustre: server umount lustre-OST0000 complete [ 3629.024821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3629.030609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.046738] Lustre: Skipped 1 previous similar message [ 3629.054358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3629.057714] Lustre: Skipped 1 previous similar message [ 3631.584486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3631.585433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3635.381832] Lustre: server umount lustre-MDT0000 complete [ 3635.698665] LustreError: 91833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3635.734254] LustreError: 91833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3639.853080] LustreError: 91817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785454843 with bad export cookie 7667392517555808158 [ 3639.854509] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3639.858788] LustreError: 91817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3640.134794] Lustre: server umount lustre-MDT0001 complete [ 3658.601631] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 3662.967835] Key type lgssc unregistered [ 3663.476622] LNet: 95421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3663.486282] LNetError: 95421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3663.518317] LNet: Removed LNI 192.168.201.122@tcp [ 3664.997165] Key type .llcrypt unregistered [ 3665.000532] Key type ._llcrypt unregistered [ 3697.309223] Key type ._llcrypt registered [ 3697.310317] Key type .llcrypt registered [ 3697.475371] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 3698.808377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3698.849920] alg: No test for adler32 (adler32-zlib) [ 3700.072761] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 3700.254846] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 3701.903300] Key type lgssc registered [ 3703.808944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3715.113770] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3715.149138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3730.913110] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3736.031324] LustreError: 96810:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 3762.847196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3771.499091] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3781.680345] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3788.194778] Lustre: server umount lustre-OST0000 complete [ 3798.495560] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 19:43:20 (1785455000) [ 3805.079845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3805.708558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3810.783823] LustreError: 98451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.804042] LustreError: 98451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3811.196151] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3814.884251] LustreError: 98452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.007203] LustreError: 98451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3821.479474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3822.101079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3829.080562] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3841.303870] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3848.723674] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3857.421387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3857.676384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3867.543250] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 3875.157046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3877.946800] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3895.776439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3895.782034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.788313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3895.792020] Lustre: Skipped 1 previous similar message [ 3895.810200] Lustre: Skipped 1 previous similar message [ 3899.691469] Lustre: server umount lustre-OST0000 complete [ 3906.026144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3906.029970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3906.042030] Lustre: Skipped 1 previous similar message [ 3906.065038] Lustre: Skipped 1 previous similar message [ 3911.136883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3911.143899] Lustre: Skipped 1 previous similar message [ 3916.257325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3916.262571] Lustre: Skipped 1 previous similar message [ 3919.327344] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3919.443116] Lustre: server umount lustre-MDT0000 complete [ 3921.376612] LustreError: 98451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3921.403407] LustreError: 98451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3923.168520] LustreError: 98431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785455126 with bad export cookie 11902504641725428940 [ 3923.172503] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3923.182296] LustreError: 98431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3924.422751] Lustre: server umount lustre-MDT0001 complete [ 3940.125188] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 3943.989648] Key type lgssc unregistered [ 3944.488169] LNet: 102149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3944.506252] LNetError: 102149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3944.539298] LNet: Removed LNI 192.168.201.122@tcp [ 3945.763894] Key type .llcrypt unregistered [ 3945.768362] Key type ._llcrypt unregistered [ 3996.527843] Key type ._llcrypt registered [ 3996.529837] Key type .llcrypt registered [ 3996.652483] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 3998.135057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3998.258286] alg: No test for adler32 (adler32-zlib) [ 3999.673439] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 3999.945338] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 4001.671159] Key type lgssc registered [ 4002.906111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4012.699818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4012.742558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4014.216590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.115924] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4023.783292] LustreError: 104351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4023.823390] LustreError: 104351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4028.898509] LustreError: 104352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4030.960170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4031.712384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4037.802600] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4046.483042] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4056.615650] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4070.224913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4071.371730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4081.304875] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4084.678265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 4095.841705] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4112.352632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4112.361879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.395800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4113.592657] Lustre: server umount lustre-OST0000 complete [ 4119.009432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4119.030431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4119.078367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4120.544564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4120.548418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4125.667923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4125.670731] Lustre: Skipped 1 previous similar message [ 4129.761070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4129.765011] Lustre: Skipped 1 previous similar message [ 4132.322900] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4132.750836] Lustre: server umount lustre-MDT0000 complete [ 4134.892101] LustreError: 105475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4134.929479] LustreError: 105475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4139.139504] LustreError: 104331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785455342 with bad export cookie 17784708714194096061 [ 4139.140801] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4139.172989] LustreError: 104331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4139.674478] Lustre: server umount lustre-MDT0001 complete [ 4154.784736] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 4158.208436] Key type lgssc unregistered [ 4158.534950] LNet: 107911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4158.549200] LNetError: 107911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4158.563932] LNet: Removed LNI 192.168.201.122@tcp [ 4159.494226] Key type .llcrypt unregistered [ 4159.497299] Key type ._llcrypt unregistered [ 4188.293273] Key type ._llcrypt registered [ 4188.303377] Key type .llcrypt registered [ 4188.586719] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 4191.101392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4191.237130] alg: No test for adler32 (adler32-zlib) [ 4192.893291] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 4193.375245] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 4195.231395] Key type lgssc registered [ 4197.240324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4209.119751] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4209.164241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4224.927544] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4230.047749] LustreError: 109156:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 4256.735246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4264.995958] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4276.811054] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4285.163334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4285.571690] LustreError: MGC192.168.201.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4285.583945] LustreError: 110182:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4285.587169] LustreError: 110182:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4285.598916] LustreError: 110182:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 4285.654401] Lustre: server umount lustre-MDT0000 complete [ 4285.656888] LustreError: 110182:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4305.823315] Lustre: 108493:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785455493/real 1785455493] req@00000000d40eac0d x1872185677319552/t0(0) o400->MGC192.168.201.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785455509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4305.886136] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4305.932076] LustreError: 109169:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4306.269868] Lustre: server umount lustre-OST0000 complete [ 4318.924453] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 4331.798876] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 4392.111986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4392.335951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4392.358146] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4392.425152] Lustre: lustre-MDT0000: new disk, initializing [ 4392.485950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.511628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4398.602816] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4410.816310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4410.916545] Lustre: 112977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4410.964409] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4410.968375] Lustre: Skipped 1 previous similar message [ 4411.031721] Lustre: lustre-MDT0001: new disk, initializing [ 4411.119673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4411.165178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4411.178712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4416.577612] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4427.702669] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4434.726992] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4442.847232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4443.074966] Lustre: lustre-OST0000: new disk, initializing [ 4443.078608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4443.083352] Lustre: 114946:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4443.211791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4450.641967] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4452.459472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4452.481675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4452.620679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4461.686639] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4467.675555] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4467.952749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4474.325317] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4474.660950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4478.433472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.444622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4478.470433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4478.487529] Lustre: Skipped 1 previous similar message [ 4483.556189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4483.560272] Lustre: Skipped 1 previous similar message [ 4488.679666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4488.690497] Lustre: Skipped 1 previous similar message [ 4491.231127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4491.846295] Lustre: server umount lustre-OST0000 complete [ 4498.400103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4498.403272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.422784] Lustre: Skipped 1 previous similar message [ 4498.435861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4503.463342] Lustre: server umount lustre-MDT0000 complete [ 4507.760640] LustreError: 112969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785455711 with bad export cookie 6819444891610289863 [ 4507.774778] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4507.775406] LustreError: 112969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4508.169909] Lustre: server umount lustre-MDT0001 complete [ 4518.334799] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 19:55:20 (1785455720) [ 4522.599515] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4546.853175] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 4560.704260] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 4610.887128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4611.123923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4611.150736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4611.277923] Lustre: lustre-MDT0000: new disk, initializing [ 4611.500570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4611.553709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4616.839666] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4629.626434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4629.765323] Lustre: 119854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4629.813432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4629.822333] Lustre: Skipped 1 previous similar message [ 4629.881251] Lustre: lustre-MDT0001: new disk, initializing [ 4629.973461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4630.009990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4630.017420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4635.155806] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4646.761477] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4654.537436] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4663.988916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4664.447400] Lustre: lustre-OST0000: new disk, initializing [ 4664.453469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4664.464452] Lustre: 121823:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4664.612617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.672259] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4673.081282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4673.094841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4673.255678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4682.974074] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4689.100615] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4689.331309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4694.717746] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4695.025881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4699.110693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4699.121630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4699.131477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4699.133071] Lustre: Skipped 2 previous similar messages [ 4699.154556] Lustre: Skipped 2 previous similar messages [ 4703.527210] Lustre: server umount lustre-OST0000 complete [ 4709.344630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4709.346427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4709.362864] Lustre: Skipped 1 previous similar message [ 4714.489677] LustreError: 121018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4714.517297] LustreError: 121018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4714.641797] Lustre: server umount lustre-MDT0000 complete [ 4719.586419] LustreError: 119865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.613246] LustreError: 119865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4719.944068] LustreError: 119847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785455923 with bad export cookie 6819444891610290745 [ 4719.966627] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4720.589661] Lustre: server umount lustre-MDT0001 complete [ 4731.014666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4731.880350] LustreError: 124040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4732.005102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4737.513755] LustreError: 124041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.755099] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4742.633788] LustreError: 124040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4752.865762] LustreError: 124041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4752.887478] LustreError: 124041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4754.698964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4763.582489] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4777.460676] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4785.989806] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4794.836297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4805.740549] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4815.192894] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4823.637473] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4831.209068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4831.226687] LustreError: Skipped 1 previous similar message [ 4831.234205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4831.256251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4831.282370] Lustre: Skipped 1 previous similar message [ 4836.201844] Lustre: server umount lustre-OST0000 complete [ 4842.466623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4842.474071] LustreError: Skipped 1 previous similar message [ 4842.483448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4842.494812] Lustre: Skipped 1 previous similar message [ 4848.126920] Lustre: server umount lustre-MDT0000 complete [ 4848.617694] LustreError: 124036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.637450] LustreError: 124036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4853.444352] LustreError: 124022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785456056 with bad export cookie 6819444891610291627 [ 4853.502141] LustreError: 124022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4853.521957] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4854.209790] Lustre: server umount lustre-MDT0001 complete [ 4870.174238] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 4874.046118] Key type lgssc unregistered [ 4874.300922] LNet: 127689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4874.309671] LNetError: 127689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4874.326719] LNet: Removed LNI 192.168.201.122@tcp [ 4875.090434] Key type .llcrypt unregistered [ 4875.091939] Key type ._llcrypt unregistered [ 4898.749273] Key type ._llcrypt registered [ 4898.753056] Key type .llcrypt registered [ 4898.884888] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 4919.454517] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 4920.687440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4920.908372] alg: No test for adler32 (adler32-zlib) [ 4922.237674] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 4922.636071] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 4924.311366] Key type lgssc registered [ 4926.356317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4970.466662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4970.499106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4971.805142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4971.920634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4972.015247] Lustre: lustre-MDT0000: new disk, initializing [ 4972.097391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4972.127385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4980.702542] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 4995.779970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4995.907445] Lustre: 130846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4995.966706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4995.971610] Lustre: Skipped 1 previous similar message [ 4996.076712] Lustre: lustre-MDT0001: new disk, initializing [ 4996.214304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4996.262976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4996.283881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5001.106341] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5014.237569] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5021.325448] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5033.315381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5034.378184] Lustre: lustre-OST0000: new disk, initializing [ 5034.396916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5034.418323] Lustre: 132816:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5034.717413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5041.791879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5041.801964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5042.215360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5045.566454] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5059.349929] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5066.436744] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5066.742815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5072.547882] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5072.725543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5077.989908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5078.009590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5078.026314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5078.045736] LustreError: Skipped 1 previous similar message [ 5078.068953] Lustre: Skipped 1 previous similar message [ 5083.105271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5088.235312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5088.249288] Lustre: Skipped 2 previous similar messages [ 5089.247758] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5089.649803] Lustre: server umount lustre-OST0000 complete [ 5098.469873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5098.480941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5098.481460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5098.487163] Lustre: Skipped 2 previous similar messages [ 5098.516565] Lustre: Skipped 1 previous similar message [ 5100.653474] Lustre: server umount lustre-MDT0000 complete [ 5106.242525] LustreError: 130838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785456309 with bad export cookie 13845783110008318101 [ 5106.259348] LustreError: 130838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5106.261943] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5106.853366] Lustre: server umount lustre-MDT0001 complete [ 5117.879166] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 20:05:19 (1785456319) [ 5126.925734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5127.511786] LustreError: 135231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5127.542410] LustreError: 135231:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5127.641080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5132.781389] LustreError: 135232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5133.522598] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5136.877078] LustreError: 135231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5141.986023] LustreError: 135232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5143.061431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5143.383116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5150.598921] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5162.702298] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5169.569455] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5173.223101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5173.241670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5178.189697] Lustre: server umount lustre-MDT0000 complete [ 5178.344023] LustreError: 135232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5178.351556] LustreError: 135232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5182.501229] LustreError: 135213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785456385 with bad export cookie 13845783110008318983 [ 5182.503742] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5182.513359] LustreError: 135213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5182.951492] Lustre: server umount lustre-MDT0001 complete [ 5194.254219] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 20:06:35 (1785456395) [ 5202.395402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5218.015603] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5223.137421] LustreError: 137649:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 5248.927528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5257.323528] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5268.950710] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5271.169092] Lustre: server umount lustre-OST0000 complete [ 5285.963977] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 20:08:07 (1785456487) [ 5293.325919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5293.978805] LustreError: 139097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5294.073408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5299.012700] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5310.231248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5316.238609] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5325.288817] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5331.249586] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5339.371578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5339.631323] LustreError: 140916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5339.647436] LustreError: 140916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5339.880462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5339.887728] Lustre: Skipped 1 previous similar message [ 5347.356857] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5356.769549] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5366.762577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5366.763917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5366.786142] Lustre: Skipped 3 previous similar messages [ 5366.808809] Lustre: Skipped 3 previous similar messages [ 5371.432258] Lustre: server umount lustre-MDT0000 complete [ 5371.879757] LustreError: 141393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5371.911039] LustreError: 141393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5376.740989] LustreError: 139078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785456580 with bad export cookie 13845783110008319648 [ 5376.753165] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5376.756245] LustreError: 139078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5377.001580] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5377.378907] Lustre: server umount lustre-MDT0001 complete [ 5384.654128] Lustre: server umount lustre-OST0000 complete [ 5396.125578] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 20:09:57 (1785456597) [ 5405.510123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5406.146715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.866798] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5423.049845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5429.614505] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5442.468498] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5449.107488] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5457.898841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5466.232174] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5476.666652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 5477.003681] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5484.640878] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5485.113883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5493.492830] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5493.940238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5499.382125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5499.397721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5499.419589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5502.434075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5503.832836] Lustre: server umount lustre-OST0000 complete [ 5510.628698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5515.515639] Lustre: server umount lustre-MDT0000 complete [ 5517.793640] LustreError: 142749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5517.832214] LustreError: 142749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5521.476684] LustreError: 142733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785456724 with bad export cookie 13845783110008320719 [ 5521.488876] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5521.500683] LustreError: 142733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5522.324486] Lustre: server umount lustre-MDT0001 complete [ 5535.063525] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 20:12:16 (1785456736) [ 5544.845045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5560.479673] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5565.600714] LustreError: 146920:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 5591.330321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5591.337851] Lustre: Skipped 2 previous similar messages [ 5599.787479] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5610.069428] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5618.738337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5619.516710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5625.572397] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5637.983707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5647.212424] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5660.831665] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5675.790708] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5687.855143] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5688.252264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5693.934328] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5694.181500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5700.576948] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5700.579331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5700.608752] Lustre: Skipped 4 previous similar messages [ 5700.623388] Lustre: Skipped 7 previous similar messages [ 5703.823723] Lustre: server umount lustre-MDT0000 complete [ 5705.704990] LustreError: 146946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5705.754915] LustreError: 146946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5709.481885] LustreError: 146927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785456912 with bad export cookie 13845783110008321741 [ 5709.489122] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5709.495634] LustreError: 146927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5710.215928] Lustre: server umount lustre-MDT0001 complete [ 5726.232687] Lustre: server umount lustre-OST0000 complete [ 5737.405463] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 20:15:38 (1785456938) [ 5746.144973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5761.887862] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5767.008179] LustreError: 151122:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 5800.673111] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5810.038931] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5817.751824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5818.641918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5824.119640] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5835.808096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5842.692527] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5852.441437] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5860.861754] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5870.339305] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5870.515843] Lustre: lustre-OST0001: new disk, initializing [ 5870.526941] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5870.536647] Lustre: 153996:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5870.710973] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5870.719421] Lustre: Skipped 5 previous similar messages [ 5877.857988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5877.862638] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5878.082472] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5881.003847] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 5894.613779] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5900.087263] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5900.319275] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5906.203911] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5906.468158] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5910.496657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5910.506648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5910.532336] Lustre: Skipped 1 previous similar message [ 5910.545820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5915.527506] Lustre: server umount lustre-OST0000 complete [ 5927.434903] Lustre: server umount lustre-OST0001 complete [ 5933.536514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5933.540745] LustreError: Skipped 3 previous similar messages [ 5939.746513] Lustre: server umount lustre-MDT0000 complete [ 5944.329391] LustreError: 153995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785457147 with bad export cookie 13845783110008322693 [ 5944.344695] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5944.360745] LustreError: 153995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5945.450548] Lustre: server umount lustre-MDT0001 complete [ 5977.047673] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 20:19:38 (1785457178) [ 5978.699651] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5981.227988] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5983.372308] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 20:19:45 (1785457185) [ 5991.574720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5991.818722] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5991.839479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5996.117160] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6007.576611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6007.671296] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6007.696350] Lustre: 157562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6013.702786] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6025.049908] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6035.488858] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6045.458784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6045.605904] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6051.628628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 6054.158877] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6063.666317] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6070.123849] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6070.529040] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6075.640308] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6075.782495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6082.533694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6082.540508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.550402] LustreError: Skipped 1 previous similar message [ 6082.550786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6082.550790] Lustre: Skipped 8 previous similar messages [ 6082.594208] Lustre: Skipped 6 previous similar messages [ 6092.255497] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6092.536435] Lustre: server umount lustre-OST0000 complete [ 6119.997132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6127.492729] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6136.714793] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6137.834381] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.841595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.845625] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6142.470845] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6142.815608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6148.722811] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6148.963457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6155.444363] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c26a004e800.ost_server_uuid 50 [ 6157.412288] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8c26a004e800.ost_server_uuid in FULL state after 0 sec [ 6157.881521] LustreError: 157569:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.22@tcp) returned error from blocking AST (req@000000002ee16f08 x1872186442033536 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000fb676f52/0xc0261a61652643df lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.22@tcp remote: 0x497ed8eac98f13ee expref: 5 pid: 158903 timeout: 6257 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6157.935490] LustreError: lustre-MDT0000: A client on nid 192.168.201.22@tcp was evicted due to a lock blocking callback time out: rc -107 [ 6157.961599] LustreError: 157558:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.22@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000fb676f52/0xc0261a61652643df lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.22@tcp remote: 0x497ed8eac98f13ee expref: 6 pid: 158903 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6168.660471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6168.672205] LustreError: Skipped 1 previous similar message [ 6172.143422] Lustre: server umount lustre-OST0000 complete [ 6184.497177] Lustre: server umount lustre-MDT0000 complete [ 6188.957267] LustreError: 157554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785457392 with bad export cookie 13845783110008323869 [ 6188.963294] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6188.971113] LustreError: 157554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6203.359909] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6203.713993] Lustre: server umount lustre-MDT0001 complete [ 6217.817855] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 6221.808605] Key type lgssc unregistered [ 6222.288525] LNet: 163687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6222.298873] LNetError: 163687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6222.330834] LNet: Removed LNI 192.168.201.122@tcp [ 6223.461395] Key type .llcrypt unregistered [ 6223.464443] Key type ._llcrypt unregistered [ 6253.460711] Key type ._llcrypt registered [ 6253.467614] Key type .llcrypt registered [ 6253.627511] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 6255.520843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6255.567682] alg: No test for adler32 (adler32-zlib) [ 6256.849758] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 6257.165843] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 6258.895218] Key type lgssc registered [ 6260.865227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6271.943942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6271.976143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6273.825442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6281.485360] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6284.267317] LustreError: 165104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6284.288289] LustreError: 165104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6289.380863] LustreError: 165105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6293.030243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6293.597604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6300.659534] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6311.698818] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6319.315675] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6329.124838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6329.768849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6340.642758] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6344.214711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6352.897336] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6360.671407] Lustre: Failing over lustre-MDT0000 [ 6361.003095] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.22@tcp (stopping) [ 6361.236192] Lustre: server umount lustre-MDT0000 complete [ 6364.641684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.659528] Lustre: Skipped 2 previous similar messages [ 6364.662165] LustreError: 165100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6364.685719] LustreError: 165100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6369.762923] LustreError: 165833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6369.795103] LustreError: 165833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6374.729237] LustreError: 165085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.22@tcp arrived at 1785457578 with bad export cookie 6182909247176188872 [ 6374.883962] LustreError: 165105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6374.919971] LustreError: 165105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6380.001709] Lustre: 164415:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785457567/real 1785457567] req@0000000085d302c5 x1872187841609856/t0(0) o400->MGC192.168.201.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785457583 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6380.031655] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6384.773526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6385.124750] LustreError: 165100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6385.137110] LustreError: 165100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6390.978032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6391.103554] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6396.397607] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6396.420291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6400.799582] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6405.264846] Lustre: lustre-MDT0000: Denying connection for new client 7f8c6a20-840c-4bbd-ba2a-02bac8c6d666 (at 192.168.201.22@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:01 [ 6414.830940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6414.845099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.883193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6416.864261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6416.869287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6421.271422] Lustre: server umount lustre-OST0000 complete [ 6427.793376] LustreError: 168845:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 6427.801903] Lustre: 168192:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 6427.809220] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 6427.842840] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 6427.855555] LustreError: 168192:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000080b75bc2 x1872187841631872/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6427.870303] LustreError: 168192:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 6427.881492] LustreError: 168192:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 6427.897162] Lustre: lustre-MDT0000: Recovery over after 0:31, of 2 clients 0 recovered and 2 were evicted. [ 6428.286320] Lustre: server umount lustre-MDT0000 complete [ 6432.232130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6432.283310] LustreError: 167548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6432.318198] LustreError: 167548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6432.714751] LustreError: 165084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785457636 with bad export cookie 6182909247176188970 [ 6432.716151] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6432.720612] LustreError: 165084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6435.210089] Lustre: server umount lustre-MDT0001 complete [ 6448.052711] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 6451.483583] Key type lgssc unregistered [ 6451.865429] LNet: 169599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6451.899398] LNetError: 169599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6452.976447] LNet: Removed LNI 192.168.201.122@tcp [ 6454.346525] Key type .llcrypt unregistered [ 6454.351907] Key type ._llcrypt unregistered [ 6486.072522] Key type ._llcrypt registered [ 6486.073943] Key type .llcrypt registered [ 6486.221972] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 6487.634900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6487.664127] alg: No test for adler32 (adler32-zlib) [ 6488.970148] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 6489.321651] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 6491.079190] Key type lgssc registered [ 6493.436205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6504.001537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6504.030074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6505.905592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6511.651426] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6516.199454] LustreError: 171011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6516.224390] LustreError: 171011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6521.334907] LustreError: 171012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6521.488727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6522.139443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6529.344560] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6540.345982] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6547.643204] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6557.819645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6558.378722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6567.030768] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6575.987170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6581.211084] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6591.467454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6591.479077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6591.485034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6591.506640] Lustre: Skipped 1 previous similar message [ 6596.601272] LustreError: 172835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6596.830965] Lustre: server umount lustre-OST0000 complete [ 6604.259952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6604.267583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6604.272316] Lustre: Skipped 1 previous similar message [ 6604.276076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6606.825038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6606.842953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6608.958263] Lustre: server umount lustre-MDT0000 complete [ 6611.950268] LustreError: 171932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6611.989419] LustreError: 171932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6613.713896] LustreError: 170992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785457817 with bad export cookie 18322305824398165936 [ 6613.718509] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6613.740701] LustreError: 170992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6614.108474] Lustre: server umount lustre-MDT0001 complete [ 6629.311581] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 6632.965041] Key type lgssc unregistered [ 6633.450669] LNet: 174546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6633.458470] LNetError: 174546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6634.537135] LNet: Removed LNI 192.168.201.122@tcp [ 6635.629140] Key type .llcrypt unregistered [ 6635.631885] Key type ._llcrypt unregistered [ 6680.761186] Key type ._llcrypt registered [ 6680.764779] Key type .llcrypt registered [ 6680.869380] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 6682.729790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6682.758851] alg: No test for adler32 (adler32-zlib) [ 6683.781476] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 6684.025843] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 6685.745982] Key type lgssc registered [ 6687.265788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6698.759581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6698.786229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6700.288852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6705.176257] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6709.728798] LustreError: 176254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.738026] LustreError: 176254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6714.849346] LustreError: 176255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6717.087886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.515062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6722.153744] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6734.250754] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6743.078471] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6751.718329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6752.120771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6759.893044] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6767.775440] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6770.554426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 6786.088730] Lustre: server umount lustre-OST0000 complete [ 6794.208587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6794.215217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6794.242488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6796.256937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6796.259629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6801.382053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6801.392573] Lustre: Skipped 2 previous similar messages [ 6802.463769] Lustre: 175565:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785457989/real 1785457989] req@00000000f6aea9e2 x1872188289351040/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785458005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6802.525301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6803.785026] Lustre: server umount lustre-MDT0000 complete [ 6804.480335] Lustre: 175564:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785457991/real 1785457991] req@000000000cd11ee2 x1872188289351424/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785458007 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6807.583481] Lustre: 175564:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785457994/real 1785457994] req@000000006c66230a x1872188289352448/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785458010 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6808.038316] LustreError: 176983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6808.066937] LustreError: 176983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6808.254596] LustreError: 176236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785458011 with bad export cookie 4964725398796569311 [ 6808.260425] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6808.272711] LustreError: 176236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6808.688794] Lustre: server umount lustre-MDT0001 complete [ 6824.491274] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 6828.256452] Key type lgssc unregistered [ 6828.488170] LNet: 179812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6828.491173] LNetError: 179812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6828.509190] LNet: Removed LNI 192.168.201.122@tcp [ 6829.493186] Key type .llcrypt unregistered [ 6829.494940] Key type ._llcrypt unregistered [ 6864.395276] Key type ._llcrypt registered [ 6864.398885] Key type .llcrypt registered [ 6864.555767] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 6866.023759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6866.192592] alg: No test for adler32 (adler32-zlib) [ 6867.387234] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 6867.622483] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 6869.295228] Key type lgssc registered [ 6870.958373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6885.386662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6885.410551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6886.976324] Lustre: *** cfs_fail_loc=135, val=0*** [ 6887.165682] LustreError: 181338:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6887.176305] LustreError: 181338:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.122@tcp: cfg command failed: rc = -2 [ 6887.186690] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6887.186690] [ 6887.201865] LustreError: MGC192.168.201.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6887.224499] LustreError: 181299:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6887.233239] LustreError: 181299:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 6887.243640] LustreError: 181299:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 6887.275131] LustreError: 181299:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6887.479767] Lustre: server umount lustre-MDT0000 complete [ 6887.484877] LustreError: 181299:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6896.722370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6897.426609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6903.889648] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6906.866458] LustreError: 181711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6906.895522] LustreError: 181711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6911.971739] LustreError: 181712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6916.661764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6917.354168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6922.176925] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 6932.388592] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6939.216366] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6941.795918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6941.845379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6942.882456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6942.894056] Lustre: Skipped 2 previous similar messages [ 6948.300359] Lustre: server umount lustre-MDT0000 complete [ 6948.322389] LustreError: 181706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6948.352978] LustreError: 181706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6953.309687] LustreError: 181692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785458156 with bad export cookie 16925419586452445197 [ 6953.328600] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6953.866787] Lustre: server umount lustre-MDT0001 complete [ 6968.962699] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 6972.778397] Key type lgssc unregistered [ 6973.159517] LNet: 184031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6973.169451] LNetError: 184031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6973.181285] LNet: Removed LNI 192.168.201.122@tcp [ 6974.428516] Key type .llcrypt unregistered [ 6974.430240] Key type ._llcrypt unregistered [ 7003.866775] Key type ._llcrypt registered [ 7003.871750] Key type .llcrypt registered [ 7004.106417] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 7004.735721] Key type .llcrypt unregistered [ 7004.740238] Key type ._llcrypt unregistered [ 7027.700239] Key type ._llcrypt registered [ 7027.702933] Key type .llcrypt registered [ 7027.818167] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 7028.992122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7029.025809] alg: No test for adler32 (adler32-zlib) [ 7030.153710] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 7030.356377] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 7031.975237] Key type lgssc registered [ 7032.813667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7042.547352] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7042.567165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7058.271926] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7063.391302] LustreError: 186166:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.122@tcp: failed processing log, type 4: rc = -110 [ 7090.143582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7100.873549] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7110.717725] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7119.461983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7120.150924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7120.281029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 7125.239945] LustreError: 186193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7125.296487] LustreError: 186193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7126.009895] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7130.608574] LustreError: 187207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7130.643237] LustreError: 187207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7135.729494] LustreError: 187206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7135.754641] LustreError: 187206:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7136.617975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.398541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7143.739609] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7152.864685] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7159.430608] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7163.196345] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7176.672539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7176.677839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.711254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7178.210516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7178.218717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7178.230043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7178.252996] Lustre: Skipped 1 previous similar message [ 7180.041755] Lustre: server umount lustre-OST0000 complete [ 7188.451562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7188.453680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7188.489556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7188.497719] Lustre: Skipped 1 previous similar message [ 7190.953199] Lustre: server umount lustre-MDT0000 complete [ 7193.578803] LustreError: 187210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7193.616671] LustreError: 187210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7195.523223] LustreError: 186173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785458398 with bad export cookie 10980351362832660624 [ 7195.541114] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7196.224380] Lustre: server umount lustre-MDT0001 complete [ 7212.081709] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 7216.542123] Key type lgssc unregistered [ 7216.985727] LNet: 190085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7216.989429] LNetError: 190085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7217.012880] LNet: Removed LNI 192.168.201.122@tcp [ 7218.173262] Key type .llcrypt unregistered [ 7218.174595] Key type ._llcrypt unregistered [ 7249.505659] Key type ._llcrypt registered [ 7249.508741] Key type .llcrypt registered [ 7249.656171] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 7250.742057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7250.885501] alg: No test for adler32 (adler32-zlib) [ 7252.159808] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 7252.375728] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 7254.087242] Key type lgssc registered [ 7255.414406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7265.848904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7265.865762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.370239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7272.687811] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7277.543249] LustreError: 191497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7277.574524] LustreError: 191497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7282.670480] LustreError: 191498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7283.797298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7284.903533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7291.800340] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7303.777673] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7314.298097] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7322.113570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7322.407733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7329.748274] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7336.790593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 7339.590028] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7346.228577] Lustre: Failing over lustre-MDT0000 [ 7346.770932] Lustre: server umount lustre-MDT0000 complete [ 7347.172407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7347.178666] LustreError: 191494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7347.201806] Lustre: Skipped 2 previous similar messages [ 7347.225222] LustreError: 191494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7352.288983] LustreError: 191491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7352.319744] LustreError: 191491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7357.410650] LustreError: 191497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7357.431960] LustreError: 191497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7363.552463] Lustre: 190809:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785458550/real 1785458550] req@00000000efb2abd0 x1872188884939776/t0(0) o400->MGC192.168.201.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785458566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7363.598851] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7368.043746] LustreError: 191491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7368.062779] LustreError: 191491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7368.085821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7372.768354] LustreError: 190807:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000073376f6f x1872188884945280/t0(0) o250->MGC192.168.201.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7373.106246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7373.166619] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7374.207948] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7378.112629] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7378.411369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7378.449327] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7378.531492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 7384.180624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 7386.909332] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 7397.346724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7397.349083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7397.378318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7398.909899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7398.911907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7402.109601] Lustre: server umount lustre-OST0000 complete [ 7409.120510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7409.139603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7409.140743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7409.140749] Lustre: Skipped 1 previous similar message [ 7409.142245] Lustre: Skipped 1 previous similar message [ 7413.733843] Lustre: server umount lustre-MDT0000 complete [ 7414.240824] LustreError: 191493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7414.266047] LustreError: 191493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 7417.980260] LustreError: 191478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785458621 with bad export cookie 6779680364114625960 [ 7417.991341] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7417.998024] LustreError: 191478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7418.412208] Lustre: server umount lustre-MDT0001 complete [ 7433.566871] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 7436.958984] Key type lgssc unregistered [ 7437.352830] LNet: 196247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7437.358969] LNetError: 196247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7437.390415] LNet: Removed LNI 192.168.201.122@tcp [ 7438.728395] Key type .llcrypt unregistered [ 7438.730308] Key type ._llcrypt unregistered [ 7478.633813] Key type ._llcrypt registered [ 7478.645127] Key type .llcrypt registered [ 7478.838935] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 7480.235191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7480.381348] alg: No test for adler32 (adler32-zlib) [ 7481.732407] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 7482.199918] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 7483.943185] Key type lgssc registered [ 7485.867983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7495.670271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7495.717906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.528515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7502.954984] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7506.932690] LustreError: 197661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7506.957833] LustreError: 197661:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7512.035424] LustreError: 197660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7516.251624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7516.671831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7524.575481] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7535.009120] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7542.075994] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7550.203725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7550.704046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7557.960227] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7567.123944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 7568.366821] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7572.672654] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7578.901767] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 7598.125402] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7607.264735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7607.277281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7607.290845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7608.292412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7608.294031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7608.295114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7612.981401] Lustre: server umount lustre-OST0000 complete [ 7618.530313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7618.534671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7618.547195] Lustre: Skipped 1 previous similar message [ 7618.574054] Lustre: Skipped 2 previous similar messages [ 7623.351754] Lustre: server umount lustre-MDT0000 complete [ 7623.649458] LustreError: 197661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7623.658760] LustreError: 197661:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7627.800031] LustreError: 197642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785458831 with bad export cookie 16213989732646296049 [ 7627.822653] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7627.822943] LustreError: 197642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7629.002835] Lustre: server umount lustre-MDT0001 complete [ 7649.967916] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 7653.469583] Key type lgssc unregistered [ 7653.845812] LNet: 201553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7653.852780] LNetError: 201553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7653.871023] LNet: Removed LNI 192.168.201.122@tcp [ 7655.116762] Key type .llcrypt unregistered [ 7655.122233] Key type ._llcrypt unregistered [ 7680.104481] Key type ._llcrypt registered [ 7680.107872] Key type .llcrypt registered [ 7680.335365] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 7681.871084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7681.975746] alg: No test for adler32 (adler32-zlib) [ 7682.989659] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 7683.272605] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 7684.959192] Key type lgssc registered [ 7686.681195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7696.457782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7696.500346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7698.340199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7704.249233] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7708.644459] LustreError: 202828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7708.676107] LustreError: 202828:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7713.766956] LustreError: 202829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7715.397348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7715.826116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7720.304302] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7729.397333] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7736.878156] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7745.747394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7746.267812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7753.660393] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7763.024413] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7767.792894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 7774.279981] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 7787.566753] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7792.473535] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 7799.321439] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 7813.094590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7813.103868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.123324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7814.124217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7814.136093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7814.147986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7819.242567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7819.251219] Lustre: Skipped 2 previous similar messages [ 7824.353557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7824.362901] Lustre: Skipped 1 previous similar message [ 7826.399328] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7826.535920] Lustre: server umount lustre-OST0000 complete [ 7833.568645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7833.578403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7833.615388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7839.671816] Lustre: server umount lustre-MDT0000 complete [ 7839.718559] LustreError: 202829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7839.728224] LustreError: 202829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7844.745220] LustreError: 205509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785459048 with bad export cookie 5470199936479753629 [ 7844.752923] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7844.765218] LustreError: 205509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7845.193899] Lustre: server umount lustre-MDT0001 complete [ 7859.149151] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 7862.076310] Key type lgssc unregistered [ 7862.544219] LNet: 207610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7862.556709] LNetError: 207610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7862.592911] LNet: Removed LNI 192.168.201.122@tcp [ 7863.581638] Key type .llcrypt unregistered [ 7863.587721] Key type ._llcrypt unregistered [ 7893.920867] Key type ._llcrypt registered [ 7893.927107] Key type .llcrypt registered [ 7894.080896] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 7895.409960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7895.527375] alg: No test for adler32 (adler32-zlib) [ 7896.605460] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 7896.853769] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 7898.495614] Key type lgssc registered [ 7899.642170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7910.285597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7910.300669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7911.916570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7917.288382] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7922.152260] LustreError: 209025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7922.168522] LustreError: 209025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7927.268194] LustreError: 209026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7927.918370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7928.526573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7934.443095] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7945.107216] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7953.657427] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7961.983668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7962.558833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7971.168717] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 7982.023601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7982.117089] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8003.592676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8003.602165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8003.632906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8007.653990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8007.654831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8007.673493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8007.716323] Lustre: Skipped 1 previous similar message [ 8009.650374] Lustre: server umount lustre-OST0000 complete [ 8017.889865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.928489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8017.934923] Lustre: Skipped 1 previous similar message [ 8023.009404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8023.016305] Lustre: Skipped 1 previous similar message [ 8028.128810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8028.134681] Lustre: Skipped 1 previous similar message [ 8030.177080] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8030.757091] Lustre: server umount lustre-MDT0000 complete [ 8033.249136] LustreError: 209020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8033.285647] LustreError: 209020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8037.883365] LustreError: 210874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785459241 with bad export cookie 4979486566924237710 [ 8037.888264] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8037.910245] LustreError: 210874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8038.585523] Lustre: server umount lustre-MDT0001 complete [ 8054.696625] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 8059.469828] Key type lgssc unregistered [ 8059.979312] LNet: 212715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8059.987347] LNetError: 212715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8060.013870] LNet: Removed LNI 192.168.201.122@tcp [ 8061.630170] Key type .llcrypt unregistered [ 8061.632517] Key type ._llcrypt unregistered [ 8087.478858] Key type ._llcrypt registered [ 8087.481275] Key type .llcrypt registered [ 8087.585838] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 8089.053993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8089.089068] alg: No test for adler32 (adler32-zlib) [ 8090.299646] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 8090.802242] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 8092.583163] Key type lgssc registered [ 8094.086522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8103.206754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8103.232869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8105.231203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8113.237232] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8115.689763] LustreError: 213984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8115.728964] LustreError: 213984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8120.802918] LustreError: 213985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8125.929670] LustreError: 213984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8127.025956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8127.919403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8134.016472] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8146.539462] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8155.633924] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8166.351829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8167.006492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8174.922273] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8176.291305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 8189.986652] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8212.462894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8212.465103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8212.479264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8212.522693] Lustre: Skipped 1 previous similar message [ 8217.571812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8217.582224] Lustre: Skipped 2 previous similar messages [ 8218.863739] Lustre: server umount lustre-OST0000 complete [ 8225.250767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8225.265360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8225.335264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8227.813831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8227.820663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8230.249689] Lustre: server umount lustre-MDT0000 complete [ 8232.934614] LustreError: 213984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8232.953271] LustreError: 213984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8236.221993] LustreError: 213965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785459439 with bad export cookie 17634176399162079950 [ 8236.235995] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8236.236472] LustreError: 213965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8236.658447] Lustre: server umount lustre-MDT0001 complete [ 8249.881073] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 8252.714816] Key type lgssc unregistered [ 8253.046866] LNet: 217673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8253.066404] LNetError: 217673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8253.083390] LNet: Removed LNI 192.168.201.122@tcp [ 8254.209105] Key type .llcrypt unregistered [ 8254.219464] Key type ._llcrypt unregistered [ 8280.624432] Key type ._llcrypt registered [ 8280.628246] Key type .llcrypt registered [ 8280.866682] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 8282.594422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8282.644382] alg: No test for adler32 (adler32-zlib) [ 8283.915553] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 8284.339766] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 8286.047178] Key type lgssc registered [ 8287.753509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8297.364564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8297.389029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8298.962311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8304.825385] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8309.218827] LustreError: 218944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8309.243969] LustreError: 218944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8314.347140] LustreError: 218945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8315.685573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8316.227964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8322.088905] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8332.258321] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8340.030220] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8349.237885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8350.036864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8358.629199] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8368.860192] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8369.535660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 8380.925189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8381.081702] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8381.512437] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8390.683688] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8401.220633] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8403.187322] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8415.316614] Lustre: Permanently deactivating lustre-OST0001 [ 8415.329990] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 8425.575413] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 8429.805791] Lustre: Permanently reactivating lustre-OST0001 [ 8429.811590] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 8429.814502] Lustre: Skipped 2 previous similar messages [ 8435.877077] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8435.899710] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 8435.914000] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8435.943045] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8435.996911] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8445.410611] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8445.418292] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8445.435853] Lustre: Skipped 2 previous similar messages [ 8445.457119] Lustre: Skipped 1 previous similar message [ 8450.548146] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8450.553957] Lustre: Skipped 1 previous similar message [ 8451.636688] Lustre: server umount lustre-OST0001 complete [ 8455.657394] LustreError: 222160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8455.668085] LustreError: 222160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8456.682723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8456.687402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8456.699293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8459.745724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8459.753462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8459.771853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8460.798542] LustreError: 220769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8460.817564] LustreError: 220769:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8462.752261] Lustre: server umount lustre-OST0000 complete [ 8469.988458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8469.999432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8470.067208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8470.069486] Lustre: Skipped 2 previous similar messages [ 8475.541862] Lustre: server umount lustre-MDT0000 complete [ 8476.134590] LustreError: 218944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8476.170713] LustreError: 218944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8482.535938] LustreError: 218926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785459685 with bad export cookie 8037403709068325821 [ 8482.560829] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8483.745377] Lustre: server umount lustre-MDT0001 complete [ 8499.931432] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 8503.563798] Key type lgssc unregistered [ 8503.767615] LNet: 224029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8503.781847] LNetError: 224029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8503.825606] LNet: Removed LNI 192.168.201.122@tcp [ 8504.912862] Key type .llcrypt unregistered [ 8504.914522] Key type ._llcrypt unregistered [ 8567.009589] Key type ._llcrypt registered [ 8567.010963] Key type .llcrypt registered [ 8567.169166] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 8569.006616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8569.041659] alg: No test for adler32 (adler32-zlib) [ 8570.290536] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 8570.750773] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 8572.447147] Key type lgssc registered [ 8573.609081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8583.705590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8583.736416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8585.230419] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8585.276246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8585.625520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8591.345524] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8602.214689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8602.328600] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8602.378328] Lustre: 226238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8602.744048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8607.944299] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8616.465630] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8624.425549] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8633.856347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8634.252836] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8634.516661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8642.499599] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8642.683184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 8652.522600] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8655.678032] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 8664.706452] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 8672.431952] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 8681.137306] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 8689.559202] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 8711.598035] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 8711.608286] Lustre: Skipped 2 previous similar messages [ 8747.630162] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 8747.642527] Lustre: Skipped 4 previous similar messages [ 8820.691149] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 8820.702078] Lustre: Skipped 8 previous similar messages [ 8831.968406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8831.972417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8831.983902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8832.484894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8832.513099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8832.554069] Lustre: Skipped 1 previous similar message [ 8837.608330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8837.621184] Lustre: Skipped 1 previous similar message [ 8842.723809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8846.305366] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8846.457519] Lustre: server umount lustre-OST0000 complete [ 8858.085128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8858.089480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8858.105901] Lustre: Skipped 1 previous similar message [ 8858.112542] Lustre: Skipped 2 previous similar messages [ 8860.117405] Lustre: server umount lustre-MDT0000 complete [ 8865.503088] LustreError: 226228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785460068 with bad export cookie 5748592507667129847 [ 8865.512325] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8865.520348] LustreError: 226228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8866.247387] Lustre: server umount lustre-MDT0001 complete [ 8880.860506] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 8883.882578] Key type lgssc unregistered [ 8884.235631] LNet: 231038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8884.239950] LNetError: 231038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8885.302785] LNet: Removed LNI 192.168.201.122@tcp [ 8886.622132] Key type .llcrypt unregistered [ 8886.623689] Key type ._llcrypt unregistered [ 8924.375487] Key type ._llcrypt registered [ 8924.378256] Key type .llcrypt registered [ 8924.522445] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 8926.660694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8926.701839] alg: No test for adler32 (adler32-zlib) [ 8928.090846] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 8928.645620] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 8930.439169] Key type lgssc registered [ 8932.171221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8944.207431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8944.222390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8945.641745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8949.709299] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8954.858986] LustreError: 232458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8954.883697] LustreError: 232458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8959.980777] LustreError: 232457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8960.996953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8961.607870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8968.755037] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 8980.482573] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8988.176983] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8997.760311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8998.295742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9006.981754] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_default_debug -1 all [ 9014.717545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 9015.387573] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9042.411378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9042.424226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9042.452775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9043.603937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9043.620305] Lustre: Skipped 2 previous similar messages [ 9044.750900] Lustre: server umount lustre-OST0000 complete [ 9053.664820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9053.678728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9053.699255] Lustre: Skipped 1 previous similar message [ 9053.709979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9055.200656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9056.813667] Lustre: server umount lustre-MDT0000 complete [ 9060.321670] LustreError: 232457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9060.344615] LustreError: 232457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9062.250285] LustreError: 232438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785460265 with bad export cookie 3268262009626751454 [ 9062.253548] LustreError: MGC192.168.201.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9062.266663] LustreError: 232438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9062.671654] Lustre: server umount lustre-MDT0001 complete [ 9078.167832] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 9082.710675] Key type lgssc unregistered [ 9083.194840] LNet: 236182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9083.212255] LNetError: 236182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9084.277288] LNet: Removed LNI 192.168.201.122@tcp [ 9085.770448] Key type .llcrypt unregistered [ 9085.771917] Key type ._llcrypt unregistered [ 9119.607474] Key type ._llcrypt registered [ 9119.611498] Key type .llcrypt registered [ 9119.723369] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 9120.835173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9120.901659] alg: No test for adler32 (adler32-zlib) [ 9121.939439] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 9122.204688] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 9123.887794] Key type lgssc registered [ 9125.762705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9404.524665] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 9408.788309] Key type lgssc unregistered [ 9409.218704] LNet: 238216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9409.235424] LNetError: 238216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9409.275944] LNet: Removed LNI 192.168.201.122@tcp [ 9410.436329] Key type .llcrypt unregistered [ 9410.439346] Key type ._llcrypt unregistered [ 9445.250333] Key type ._llcrypt registered [ 9445.253789] Key type .llcrypt registered [ 9445.504868] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing set_hostid [ 9465.448859] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing load_modules_local [ 9466.959022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9466.980136] alg: No test for adler32 (adler32-zlib) [ 9468.098598] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 9468.259459] LNet: Added LNI 192.168.201.122@tcp [8/256/0/180] [ 9469.967618] Key type lgssc registered [ 9471.874124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9514.081480] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9130 sec ========== 21:18:35 (1785460715) [ 9517.076381] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:18:37 (1785460717) === [ 9522.741021] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:18:43 (1785460723) === [ 9584.808191] Lustre: DEBUG MARKER: oleg122-server.virtnet: executing unload_modules_local [ 9588.373518] Key type lgssc unregistered [ 9588.735466] LNet: 243463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9588.754671] LNetError: 243463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9588.774485] LNet: Removed LNI 192.168.201.122@tcp [ 9589.818291] Key type .llcrypt unregistered [ 9589.821434] Key type ._llcrypt unregistered