[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479205133 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002365] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009137] LSM: Security Framework initializing [ 0.010064] Yama: becoming mindful. [ 0.011039] SELinux: Initializing. [ 0.012069] *** VALIDATE selinux *** [ 0.020385] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025510] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027139] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028114] *** VALIDATE tmpfs *** [ 0.029476] *** VALIDATE proc *** [ 0.030258] *** VALIDATE cgroup *** [ 0.031013] *** VALIDATE cgroup2 *** [ 0.032312] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035049] Spectre V2 : User space: Vulnerable [ 0.036010] Speculative Store Bypass: Vulnerable [ 0.038756] debug: unmapping init [mem 0xffffffffa2659000-0xffffffffa2660fff] [ 0.041168] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042778] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043026] ... version: 2 [ 0.044015] ... bit width: 48 [ 0.045014] ... generic registers: 4 [ 0.046014] ... value mask: 0000ffffffffffff [ 0.047016] ... max period: 00007fffffffffff [ 0.048015] ... fixed-purpose events: 3 [ 0.049013] ... event mask: 000000070000000f [ 0.051235] rcu: Hierarchical SRCU implementation. [ 0.053458] smp: Bringing up secondary CPUs ... [ 0.054617] x86: Booting SMP configuration: [ 0.055033] .... node #0, CPUs: #1 #2 #3 [ 0.058490] smp: Brought up 1 node, 4 CPUs [ 0.060012] smpboot: Max logical packages: 1 [ 0.061019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134022] node 0 deferred pages initialised in 71ms [ 0.138093] devtmpfs: initialized [ 0.139232] x86/mm: Memory block size: 128MB [ 0.142958] gcov: version magic: 0x41383552 [ 0.146277] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152260] pinctrl core: initialized pinctrl subsystem [ 0.155210] [ 0.155875] ************************************************************* [ 0.158014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161015] ** ** [ 0.164015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166012] ** ** [ 0.169014] ** This means that this kernel is built to expose internal ** [ 0.172015] ** IOMMU data structures, which may compromise security on ** [ 0.174012] ** your system. ** [ 0.177017] ** ** [ 0.180015] ** If you see this message and you are not debugging the ** [ 0.183014] ** kernel, report this immediately to your vendor! ** [ 0.185014] ** ** [ 0.188013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191013] ************************************************************* [ 0.193607] NET: Registered protocol family 16 [ 0.195298] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.198059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.201069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.206011] cpuidle: using governor menu [ 0.208000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.210466] PCI: Using configuration type 1 for base access [ 0.213115] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.223158] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230040] cryptd: max_cpu_qlen set to 1000 [ 0.234260] ACPI: Added _OSI(Module Device) [ 0.235016] ACPI: Added _OSI(Processor Device) [ 0.237017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.239016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.244398] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.251331] ACPI: Interpreter enabled [ 0.253053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.255016] ACPI: Using IOAPIC for interrupt routing [ 0.256143] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260433] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.271008] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.273041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.275019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.278081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.282286] acpiphp: Slot [2] registered [ 0.284106] acpiphp: Slot [5] registered [ 0.285099] acpiphp: Slot [6] registered [ 0.286150] acpiphp: Slot [7] registered [ 0.288117] acpiphp: Slot [8] registered [ 0.289089] acpiphp: Slot [9] registered [ 0.290094] acpiphp: Slot [10] registered [ 0.291161] acpiphp: Slot [3] registered [ 0.293091] acpiphp: Slot [4] registered [ 0.294066] acpiphp: Slot [11] registered [ 0.296128] acpiphp: Slot [12] registered [ 0.297113] acpiphp: Slot [13] registered [ 0.299112] acpiphp: Slot [14] registered [ 0.300103] acpiphp: Slot [15] registered [ 0.302096] acpiphp: Slot [16] registered [ 0.304122] acpiphp: Slot [17] registered [ 0.305092] acpiphp: Slot [18] registered [ 0.307112] acpiphp: Slot [19] registered [ 0.308085] acpiphp: Slot [20] registered [ 0.310105] acpiphp: Slot [21] registered [ 0.311104] acpiphp: Slot [22] registered [ 0.313116] acpiphp: Slot [23] registered [ 0.315108] acpiphp: Slot [24] registered [ 0.316122] acpiphp: Slot [25] registered [ 0.318123] acpiphp: Slot [26] registered [ 0.320100] acpiphp: Slot [27] registered [ 0.321100] acpiphp: Slot [28] registered [ 0.323093] acpiphp: Slot [29] registered [ 0.324098] acpiphp: Slot [30] registered [ 0.325105] acpiphp: Slot [31] registered [ 0.327059] PCI host bridge to bus 0000:00 [ 0.329019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.331020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.333025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.338047] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.341022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.343173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.346245] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.349343] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.356542] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.362000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.364019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.366018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.372784] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.374820] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.378060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.380871] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.388023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.401905] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.405971] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.413113] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.421016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.428016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.445027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.454485] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.462023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.473022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.495026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.505748] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.512017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.519024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.535020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.545698] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.553026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.560000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.580021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.591793] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.599020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.604025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.625021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.634904] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.640019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.645019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.660027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.669811] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.672461] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.674465] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.676404] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.680315] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.685132] iommu: Default domain type: Passthrough [ 0.687475] SCSI subsystem initialized [ 0.689188] ACPI: bus type USB registered [ 0.691099] usbcore: registered new interface driver usbfs [ 0.693103] usbcore: registered new interface driver hub [ 0.696094] usbcore: registered new device driver usb [ 0.698185] pps_core: LinuxPPS API ver. 1 registered [ 0.699013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.702060] PTP clock support registered [ 0.704191] EDAC MC: Ver: 3.0.0 [ 0.706153] PCI: Using ACPI for IRQ routing [ 0.708982] NetLabel: Initializing [ 0.710022] NetLabel: domain hash size = 128 [ 0.712017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.714091] NetLabel: unlabeled traffic allowed by default [ 0.715159] vgaarb: loaded [ 0.717211] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.719023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.727012] clocksource: Switched to clocksource kvm-clock [ 0.839276] VFS: Disk quotas dquot_6.6.0 [ 0.841254] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.844220] *** VALIDATE ramfs *** [ 0.845650] *** VALIDATE hugetlbfs *** [ 0.848317] pnp: PnP ACPI init [ 0.850970] pnp: PnP ACPI: found 6 devices [ 0.879746] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.882592] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.884418] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.886265] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.888251] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.890245] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.892428] NET: Registered protocol family 2 [ 0.894581] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.898950] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.904697] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.910683] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.913611] TCP: Hash tables configured (established 65536 bind 65536) [ 0.916112] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.918543] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.920811] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.924037] NET: Registered protocol family 1 [ 0.926835] RPC: Registered named UNIX socket transport module. [ 0.928679] RPC: Registered udp transport module. [ 0.930070] RPC: Registered tcp transport module. [ 0.931417] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.933411] NET: Registered protocol family 44 [ 0.934697] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.936382] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.939842] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.942417] PCI: CLS 0 bytes, default 64 [ 0.944933] Unpacking initramfs... [ 2.460989] debug: unmapping init [mem 0xffff8db23cc54000-0xffff8db23ffbffff] [ 2.468362] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.470679] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.474335] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.968711] Initialise system trusted keyrings [ 2.970982] Key type blacklist registered [ 2.973109] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.983301] zbud: loaded [ 2.986737] *** VALIDATE nfs *** [ 2.988128] *** VALIDATE nfs4 *** [ 2.989931] pstore: using deflate compression [ 2.993524] Platform Keyring initialized [ 3.102717] NET: Registered protocol family 38 [ 3.104270] Key type asymmetric registered [ 3.105504] Asymmetric key parser 'x509' registered [ 3.106967] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.109577] io scheduler mq-deadline registered [ 3.111325] io scheduler kyber registered [ 3.113016] io scheduler bfq registered [ 3.115410] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.118333] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.121415] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.124477] ACPI: Power Button [PWRF] [ 3.130194] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.137537] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.162335] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.169520] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.188498] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.216616] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.247278] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.257810] Non-volatile memory driver v1.3 [ 3.259241] Linux agpgart interface v0.103 [ 3.292489] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.296101] vda: detected capacity change from 0 to 76681216 [ 3.316374] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.319968] vdb: detected capacity change from 0 to 1073741824 [ 3.336097] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.339739] vdc: detected capacity change from 0 to 2621440000 [ 3.357802] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.360438] vdd: detected capacity change from 0 to 2621440000 [ 3.378052] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.381138] vde: detected capacity change from 0 to 4294967296 [ 3.398534] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.403636] vdf: detected capacity change from 0 to 4294967296 [ 3.409245] libphy: Fixed MDIO Bus: probed [ 3.414418] usbcore: registered new interface driver usbserial_generic [ 3.416563] usbserial: USB Serial support registered for generic [ 3.418444] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.421792] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.423358] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.425522] mousedev: PS/2 mouse device common for all mice [ 3.428870] rtc_cmos 00:05: RTC can wake from S4 [ 3.431765] rtc_cmos 00:05: registered as rtc0 [ 3.433626] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.436481] intel_pstate: CPU model not supported [ 3.439842] hid: raw HID events driver (C) Jiri Kosina [ 3.442114] usbcore: registered new interface driver usbhid [ 3.444440] usbhid: USB HID core driver [ 3.444893] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.446278] drop_monitor: Initializing network drop monitor service [ 3.446421] Initializing XFRM netlink socket [ 3.446813] NET: Registered protocol family 10 [ 3.448128] Segment Routing with IPv6 [ 3.456142] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.458873] NET: Registered protocol family 17 [ 3.464865] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.467321] mpls_gso: MPLS GSO support [ 3.480533] RAS: Correctable Errors collector initialized. [ 3.482510] AVX version of gcm_enc/dec engaged. [ 3.484012] AES CTR mode by8 optimization enabled [ 3.571890] sched_clock: Marking stable (3571854474, 0)->(4505758611, -933904137) [ 3.575362] registered taskstats version 1 [ 3.577268] Loading compiled-in X.509 certificates [ 3.579134] zswap: loaded using pool lzo/zbud [ 3.605552] Key type big_key registered [ 3.619124] Key type encrypted registered [ 3.620689] ima: No TPM chip found, activating TPM-bypass! [ 3.623123] ima: Allocated hash algorithm: sha1 [ 3.624447] ima: No architecture policies found [ 3.626280] evm: Initialising EVM extended attributes: [ 3.628053] evm: security.selinux [ 3.628816] evm: security.ima [ 3.629942] evm: security.capability [ 3.631178] evm: HMAC attrs: 0x1 [ 3.633448] rtc_cmos 00:05: setting system clock to 2026-09-05 20:51:52 UTC (1788641512) [ 3.639648] debug: unmapping init [mem 0xffffffffa3603000-0xffffffffa37fffff] [ 3.642964] debug: unmapping init [mem 0xffffffffa2382000-0xffffffffa2658fff] [ 3.651109] Write protecting the kernel read-only data: 28672k [ 3.654417] debug: unmapping init [mem 0xffffffffa0a03000-0xffffffffa0bfffff] [ 3.656244] debug: unmapping init [mem 0xffffffffa1314000-0xffffffffa13fffff] [ 3.693967] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.705343] systemd[1]: Detected virtualization kvm. [ 3.707226] systemd[1]: Detected architecture x86-64. [ 3.709076] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.734391] systemd[1]: No hostname configured. [ 3.736301] systemd[1]: Set hostname to . [ 3.738438] random: systemd: uninitialized urandom read (16 bytes read) [ 3.740970] systemd[1]: Initializing machine ID from random generator. [ 3.797138] random: ln: uninitialized urandom read (6 bytes read) [ 3.882476] random: systemd: uninitialized urandom read (16 bytes read) [ 3.885257] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.889709] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.893891] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.511149] device-mapper: uevent: version 1.0.3 [ 4.513967] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System.[ 5.133373] random: fast init done [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.304680] virtio_net virtio0 ens2: renamed from eth0 [ 5.373971] scsi host0: ata_piix [ 5.383046] scsi host1: ata_piix [ 5.384926] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.387921] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.732173] dracut-initqueue[592]: RTNETLINK answers: File exists [ 10.135659] random: crng init done [ 10.137136] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.608276] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.810338] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.078725] SELinux: Disabled at runtime. [ 12.139420] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.148279] systemd[1]: Detected virtualization kvm. [ 12.150331] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.705435] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.709100] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.713842] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.718273] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.721749] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.729510] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.744328] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ 12.948633] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.263887] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.582735] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.608812] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.745394] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.764058] EDAC sbridge: Ver: 1.1.2 [ 15.531241] Key type dns_resolver registered [ 15.875218] NFS: Registering the id_resolver key type [ 15.877809] Key type id_resolver registered [ 15.879795] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg612-server login: [ 53.763310] hrtimer: interrupt took 2443389 ns [ 71.585465] libcfs: loading out-of-tree module taints kernel. [ 71.728355] Key type ._llcrypt registered [ 71.731679] Key type .llcrypt registered [ 71.949711] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 91.741321] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 93.803812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 93.826947] alg: No test for adler32 (adler32-zlib) [ 95.914305] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 97.135196] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 98.967123] Key type lgssc registered [ 100.717821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 121.114821] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 160.408594] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 174.642402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 174.673843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 175.915639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 175.968265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 176.086881] Lustre: lustre-MDT0000: new disk, initializing [ 176.168485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.183252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 181.048663] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.765582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.893650] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 194.947723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 194.961859] Lustre: Skipped 1 previous similar message [ 195.102585] Lustre: lustre-MDT0001: new disk, initializing [ 195.204977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 195.248423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 195.262206] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 199.443495] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.666918] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 214.937881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.218103] Lustre: lustre-OST0000: new disk, initializing [ 215.221386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.225705] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.319939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.328686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.341211] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 217.421466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 223.185564] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.037274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.149882] Lustre: lustre-OST0001: new disk, initializing [ 239.154511] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 239.164940] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.256119] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.368348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 246.377767] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 246.429745] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 247.243072] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.433517] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 269.298828] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 276.391745] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing check_logdir /tmp/testlogs/ [ 281.446248] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing yml_node [ 285.979628] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 288.350571] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 290.952716] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 292.582396] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 16:56:38 EDT 2026 [ 311.198547] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 312.793480] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 333.793620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 333.802852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.814716] Lustre: Skipped 3 previous similar messages [ 333.814982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.835821] Lustre: Skipped 3 previous similar messages [ 335.223970] Lustre: server umount lustre-MDT0000 complete [ 338.912661] LustreError: 7472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.934889] LustreError: 7472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 339.438950] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641848 with bad export cookie 9318475198255232140 [ 339.439604] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 339.452043] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 344.035746] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.035754] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.036600] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.059376] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 344.068776] Lustre: Skipped 1 previous similar message [ 345.873718] Lustre: server umount lustre-MDT0001 complete [ 350.986396] Lustre: server umount lustre-OST0000 complete [ 356.304552] Lustre: server umount lustre-OST0001 complete [ 367.067833] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 378.258599] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 415.916815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.139615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 416.166690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 416.263679] Lustre: lustre-MDT0000: new disk, initializing [ 416.332040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.359748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 421.591593] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 435.593142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.725831] Lustre: 15081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 435.783120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 435.787176] Lustre: Skipped 1 previous similar message [ 435.853272] Lustre: lustre-MDT0001: new disk, initializing [ 435.933751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 435.961121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 435.966483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 442.100310] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 453.227192] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 460.251756] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 468.606188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 468.855751] Lustre: lustre-OST0000: new disk, initializing [ 468.859527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 468.865431] Lustre: 17053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 468.947136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.032883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 470.043217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 470.188319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 475.348867] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 485.650262] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 490.835496] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 491.024852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 496.022707] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 496.245364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 500.713016] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.716870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.728768] Lustre: Skipped 2 previous similar messages [ 504.580180] Lustre: server umount lustre-OST0000 complete [ 510.947899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.965692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 510.980295] Lustre: Skipped 2 previous similar messages [ 515.033638] Lustre: server umount lustre-MDT0000 complete [ 516.065744] LustreError: 15090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.087507] LustreError: 15090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 518.856681] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642027 with bad export cookie 9318475198255233680 [ 518.859962] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.867432] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 519.178731] Lustre: server umount lustre-MDT0001 complete [ 523.151549] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 17:00:29 (1788642029) [ 530.800931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.276388] LustreError: 19319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.361527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.913874] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 536.552897] LustreError: 19320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.067066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.153892] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 559.658313] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 565.097167] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 572.394208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 579.198322] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 586.745822] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 597.472310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 597.483853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.492939] Lustre: Skipped 1 previous similar message [ 597.498374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 601.580559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 603.757599] Lustre: server umount lustre-OST0000 complete [ 607.719260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 607.736779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.753929] Lustre: Skipped 1 previous similar message [ 607.762941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 607.767899] Lustre: Skipped 2 previous similar messages [ 622.047130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 622.265225] Lustre: server umount lustre-MDT0000 complete [ 625.901249] LustreError: 19301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642134 with bad export cookie 9318475198255234562 [ 625.904563] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 625.908836] LustreError: 19301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 626.243713] Lustre: server umount lustre-MDT0001 complete [ 638.779901] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 642.041882] Key type lgssc unregistered [ 642.449377] LNet: 22864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.460145] LNetError: 22864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 642.493875] LNet: Removed LNI 192.168.206.112@tcp [ 643.556156] Key type .llcrypt unregistered [ 643.561836] Key type ._llcrypt unregistered [ 668.907451] Key type ._llcrypt registered [ 668.916645] Key type .llcrypt registered [ 669.047737] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 670.551567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 670.634871] alg: No test for adler32 (adler32-zlib) [ 671.826391] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 672.144717] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 673.815422] Key type lgssc registered [ 675.043760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 684.334149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 684.364744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 685.858077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 690.037399] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 695.274816] LustreError: 24275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 695.290050] LustreError: 24275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 697.775287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.206105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 702.273134] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 710.705772] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 716.515222] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 723.267968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 723.673582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.766386] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 730.878150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 738.168433] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 753.632257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 753.634083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 753.668162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 753.704883] Lustre: Skipped 1 previous similar message [ 755.570171] Lustre: server umount lustre-OST0000 complete [ 759.779916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 759.784537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 759.797772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 759.804977] Lustre: Skipped 1 previous similar message [ 763.875660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 763.878460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 766.028026] Lustre: server umount lustre-MDT0000 complete [ 768.992702] LustreError: 24271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 769.002421] LustreError: 24271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 770.167307] LustreError: 24257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642279 with bad export cookie 5033364857023216101 [ 770.168613] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 770.182925] LustreError: 24257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 770.545245] Lustre: server umount lustre-MDT0001 complete [ 783.107415] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 786.259654] Key type lgssc unregistered [ 786.599477] LNet: 28248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.609181] LNetError: 28248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.642676] LNet: Removed LNI 192.168.206.112@tcp [ 787.720694] Key type .llcrypt unregistered [ 787.724465] Key type ._llcrypt unregistered [ 812.110782] Key type ._llcrypt registered [ 812.116417] Key type .llcrypt registered [ 812.224274] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 813.324469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 813.358912] alg: No test for adler32 (adler32-zlib) [ 814.587388] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 814.911023] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 816.591902] Key type lgssc registered [ 817.687766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 825.875803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 825.889696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.449464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 832.500927] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 837.607656] LustreError: 29658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 837.632878] LustreError: 29658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 842.245305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 842.562243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 847.894637] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 856.148563] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 862.129749] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 876.633103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 877.139076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.239137] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 891.209761] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 897.677305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 908.257429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 908.267946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.271743] LustreError: Skipped 1 previous similar message [ 908.272776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 908.294133] Lustre: Skipped 1 previous similar message [ 908.317350] Lustre: Skipped 1 previous similar message [ 913.356886] Lustre: server umount lustre-OST0000 complete [ 918.500199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 918.526611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 918.536630] Lustre: Skipped 1 previous similar message [ 923.535886] Lustre: server umount lustre-MDT0000 complete [ 923.626621] LustreError: 30636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.590149] LustreError: 30391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642436 with bad export cookie 17786834169796778111 [ 927.594848] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 927.601398] LustreError: 30391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 927.945085] Lustre: server umount lustre-MDT0001 complete [ 941.254765] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 944.088549] Key type lgssc unregistered [ 944.368607] LNet: 33557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.380507] LNetError: 33557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.413579] LNet: Removed LNI 192.168.206.112@tcp [ 945.346559] Key type .llcrypt unregistered [ 945.352114] Key type ._llcrypt unregistered [ 969.635660] Key type ._llcrypt registered [ 969.639177] Key type .llcrypt registered [ 969.778777] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 971.695435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 971.768095] alg: No test for adler32 (adler32-zlib) [ 972.897523] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 973.122812] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 974.800099] Key type lgssc registered [ 976.365445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 985.489681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 985.511052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.922161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.947354] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 996.341584] LustreError: 34968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.375182] LustreError: 34968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 998.651264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.091108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1003.264031] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1008.949308] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1014.430410] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1021.205864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1021.474957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1027.854390] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1031.730561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1036.402866] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1045.998881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1046.007652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.028223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1047.008591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1047.023948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1050.279899] Lustre: server umount lustre-OST0000 complete [ 1055.205329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1055.219277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.234313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1055.238477] Lustre: Skipped 1 previous similar message [ 1057.249364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1057.257563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.802921] Lustre: server umount lustre-MDT0000 complete [ 1062.382047] LustreError: 34968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1062.412567] LustreError: 34968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1063.391181] LustreError: 34948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642572 with bad export cookie 9456646024216264379 [ 1063.396067] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1063.400508] LustreError: 34948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1063.764382] Lustre: server umount lustre-MDT0001 complete [ 1074.509424] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 1077.377949] Key type lgssc unregistered [ 1077.713428] LNet: 38500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.723446] LNetError: 38500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.744743] LNet: Removed LNI 192.168.206.112@tcp [ 1078.825795] Key type .llcrypt unregistered [ 1078.828261] Key type ._llcrypt unregistered [ 1103.730611] Key type ._llcrypt registered [ 1103.732277] Key type .llcrypt registered [ 1103.826760] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 1104.716214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1104.737081] alg: No test for adler32 (adler32-zlib) [ 1105.801662] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 1106.224274] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 1107.951187] Key type lgssc registered [ 1108.798028] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1116.320305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1116.339916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.807896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1122.066354] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1127.396758] LustreError: 39910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.410057] LustreError: 39910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1131.041578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.438851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1136.595913] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1145.108239] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1152.509312] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1161.238859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1161.708332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.769119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1169.011866] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1176.805758] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1184.279391] Lustre: lustre-OST0000: Not available for connect from 192.168.206.12@tcp (stopping) [ 1187.297855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1187.308810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1187.314656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1187.354481] Lustre: Skipped 1 previous similar message [ 1189.350902] Lustre: lustre-OST0000: Not available for connect from 192.168.206.12@tcp (stopping) [ 1189.365941] Lustre: Skipped 1 previous similar message [ 1189.877665] Lustre: server umount lustre-OST0000 complete [ 1212.897621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.909708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1212.926418] Lustre: Skipped 1 previous similar message [ 1215.316276] Lustre: server umount lustre-MDT0000 complete [ 1218.019805] LustreError: 40638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1218.042977] LustreError: 40638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1219.258134] LustreError: 39890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642728 with bad export cookie 16164745030041312863 [ 1219.268604] LustreError: 39890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1219.269167] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1225.694230] Lustre: server umount lustre-MDT0001 complete [ 1237.101234] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 1239.958511] Key type lgssc unregistered [ 1240.295380] LNet: 43621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.304490] LNetError: 43621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.318116] LNet: Removed LNI 192.168.206.112@tcp [ 1242.319406] Key type .llcrypt unregistered [ 1242.321084] Key type ._llcrypt unregistered [ 1268.879624] Key type ._llcrypt registered [ 1268.881786] Key type .llcrypt registered [ 1268.988713] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 1270.461826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1270.560836] alg: No test for adler32 (adler32-zlib) [ 1271.791690] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 1272.233581] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 1274.063279] Key type lgssc registered [ 1275.756978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1284.097166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1284.130941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.757990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.666520] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1295.332776] LustreError: 45042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.377290] LustreError: 45042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1299.402645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1299.920986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1304.587646] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1312.760374] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1319.093325] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1326.617386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1326.955301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1331.064458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1334.410932] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1343.272347] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1350.406557] Lustre: server umount lustre-MDT0000 complete [ 1351.137476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1351.159056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1351.184116] LustreError: 45038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.202213] LustreError: 45038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1354.363026] LustreError: 45022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642863 with bad export cookie 10674084083847217571 [ 1354.384907] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1354.723383] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1354.728975] LustreError: 45039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.740334] Lustre: Skipped 1 previous similar message [ 1354.745876] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1356.299175] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.12@tcp (stopping) [ 1359.840508] LustreError: 45039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.840699] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1359.862337] LustreError: 45039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1360.761174] Lustre: server umount lustre-MDT0001 complete [ 1381.709664] Lustre: server umount lustre-OST0000 complete [ 1397.747508] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 1400.154829] Key type lgssc unregistered [ 1400.420695] LNet: 48900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1400.431830] LNetError: 48900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1400.441539] LNet: Removed LNI 192.168.206.112@tcp [ 1401.138907] Key type .llcrypt unregistered [ 1401.141417] Key type ._llcrypt unregistered [ 1427.027445] Key type ._llcrypt registered [ 1427.032499] Key type .llcrypt registered [ 1427.201575] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 1428.508410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1428.597709] alg: No test for adler32 (adler32-zlib) [ 1429.843415] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 1430.123648] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 1431.832265] Key type lgssc registered [ 1433.243208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1443.138873] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1443.175203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1458.847551] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1463.967336] LustreError: 50284:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.112@tcp: failed processing log, type 4: rc = -110 [ 1490.722739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.450497] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1503.018812] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1668.107501] Lustre: server umount lustre-OST0000 complete [ 1687.713797] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 1691.382582] Key type lgssc unregistered [ 1691.679098] LNet: 51963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.691767] LNetError: 51963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.707424] LNet: Removed LNI 192.168.206.112@tcp [ 1692.589862] Key type .llcrypt unregistered [ 1692.593264] Key type ._llcrypt unregistered [ 1718.190730] Key type ._llcrypt registered [ 1718.193316] Key type .llcrypt registered [ 1718.292552] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 1719.208038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1719.434198] alg: No test for adler32 (adler32-zlib) [ 1720.509246] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 1720.802867] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 1722.519166] Key type lgssc registered [ 1723.519871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1729.955700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1729.977633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.472055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1735.488877] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1741.796567] LustreError: 53374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.818671] LustreError: 53374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1742.923048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.319703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1747.903384] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1755.482290] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1761.745134] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1768.484084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1768.897536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.700919] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1777.147463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1782.215348] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1785.312654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1785.324451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.337275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1787.371403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1787.377877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1787.390303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1790.660847] Lustre: server umount lustre-OST0000 complete [ 1794.538179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1794.552338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.582754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1794.594338] Lustre: Skipped 1 previous similar message [ 1797.600488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1797.601467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.971764] Lustre: server umount lustre-MDT0000 complete [ 1802.723369] LustreError: 53368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1802.746665] LustreError: 53368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1805.120767] LustreError: 53354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643313 with bad export cookie 12781242169334413924 [ 1805.134813] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1805.148261] LustreError: 53354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1805.653759] Lustre: server umount lustre-MDT0001 complete [ 1817.144612] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 1820.559876] Key type lgssc unregistered [ 1820.971812] LNet: 56903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.986035] LNetError: 56903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.019062] LNet: Removed LNI 192.168.206.112@tcp [ 1822.080100] Key type .llcrypt unregistered [ 1822.081917] Key type ._llcrypt unregistered [ 1848.458165] Key type ._llcrypt registered [ 1848.460622] Key type .llcrypt registered [ 1848.563632] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 1849.702805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1849.919714] alg: No test for adler32 (adler32-zlib) [ 1851.092563] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 1851.442646] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 1853.071170] Key type lgssc registered [ 1854.479844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1862.392466] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1862.442766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1878.047367] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1883.167632] LustreError: 58287:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.112@tcp: failed processing log, type 4: rc = -110 [ 1909.856726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.882890] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1923.797769] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1929.686057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.089432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.174083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1934.570596] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1935.331477] LustreError: 58312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.355699] LustreError: 58312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1940.455828] LustreError: 58313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.268938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.729742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1948.821506] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1957.160976] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1963.034173] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1966.049965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1966.064966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.100319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1966.576329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.695829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1971.705612] Lustre: Skipped 2 previous similar messages [ 1976.805564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1976.815631] Lustre: Skipped 1 previous similar message [ 1979.360146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1979.512346] Lustre: server umount lustre-OST0000 complete [ 2000.973280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.992569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2005.473127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2005.488511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.508603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.354769] Lustre: server umount lustre-MDT0000 complete [ 2010.748294] LustreError: 58295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643519 with bad export cookie 13481422719420341847 [ 2010.752263] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2010.765259] LustreError: 58295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2017.333205] Lustre: server umount lustre-MDT0001 complete [ 2029.162510] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 2031.817291] Key type lgssc unregistered [ 2032.090528] LNet: 62034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.096135] LNetError: 62034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2032.109120] LNet: Removed LNI 192.168.206.112@tcp [ 2032.904890] Key type .llcrypt unregistered [ 2032.912242] Key type ._llcrypt unregistered [ 2054.823325] Key type ._llcrypt registered [ 2054.827317] Key type .llcrypt registered [ 2054.983970] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2055.949708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2056.017145] alg: No test for adler32 (adler32-zlib) [ 2057.186390] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 2057.586879] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 2059.311320] Key type lgssc registered [ 2060.649321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2069.461451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2069.482561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.927521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.390825] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2080.228064] LustreError: 63457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.249718] LustreError: 63457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2083.889131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.238336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2088.535441] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2096.810184] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2102.256752] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2108.873566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2109.198269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.387567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2115.904957] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2124.314342] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2146.275508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2146.281850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.287341] LustreError: Skipped 1 previous similar message [ 2146.288067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2146.331327] Lustre: Skipped 1 previous similar message [ 2149.056887] Lustre: server umount lustre-OST0000 complete [ 2156.005831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2156.019273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.031477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2156.037164] Lustre: Skipped 1 previous similar message [ 2159.416951] Lustre: server umount lustre-MDT0000 complete [ 2161.642418] LustreError: 64480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.660270] LustreError: 64480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2163.204759] LustreError: 63438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643672 with bad export cookie 12958156890749156186 [ 2163.211850] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2163.216981] LustreError: 63438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2163.642712] Lustre: server umount lustre-MDT0001 complete [ 2174.622142] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 2177.119271] Key type lgssc unregistered [ 2177.400747] LNet: 67015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2177.414349] LNetError: 67015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2177.441219] LNet: Removed LNI 192.168.206.112@tcp [ 2178.403682] Key type .llcrypt unregistered [ 2178.405135] Key type ._llcrypt unregistered [ 2202.546267] Key type ._llcrypt registered [ 2202.548278] Key type .llcrypt registered [ 2202.672661] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2203.712387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2203.768400] alg: No test for adler32 (adler32-zlib) [ 2204.906158] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 2205.168342] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 2206.847638] Key type lgssc registered [ 2208.278132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2217.307213] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 17:28:43 (1788643723) [ 2224.009728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2224.032771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2225.476213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.375423] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2235.874706] LustreError: 68708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.889074] LustreError: 68708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2238.806918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.359572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2244.459842] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2252.840199] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2261.113692] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2268.800341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2269.269822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2270.339561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2276.571178] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2286.836929] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2293.986211] Lustre: Failing over lustre-MDT0000 [ 2294.195836] Lustre: server umount lustre-MDT0000 complete [ 2294.792264] LustreError: 68704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.264683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.286775] Lustre: Skipped 2 previous similar messages [ 2299.886770] LustreError: 68703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.918884] LustreError: 68703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2303.868525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2304.091053] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2304.297863] Lustre: *** cfs_fail_loc=135, val=0*** [ 2304.403759] LustreError: 71794:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2304.409684] LustreError: 71794:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.112@tcp: cfg command failed: rc = -2 [ 2304.417188] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2304.417188] [ 2304.425766] LustreError: MGC192.168.206.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2304.437154] LustreError: 71782:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2304.444329] LustreError: 71782:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2304.452625] LustreError: 71782:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2304.459990] LustreError: 71782:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2304.480407] Lustre: server umount lustre-MDT0000 complete [ 2304.486882] LustreError: 71782:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2305.007493] LustreError: 68704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.041715] LustreError: 68704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2310.133051] LustreError: 70020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.172676] LustreError: 70020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2311.489591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2311.705146] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2312.111257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2312.190026] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2316.279494] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2316.979991] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2317.299813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2317.316762] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2317.372191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2325.984200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2325.991609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2326.011689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2327.538253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2327.544803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.551561] Lustre: server umount lustre-OST0000 complete [ 2337.764622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2337.780716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2337.783551] Lustre: Skipped 1 previous similar message [ 2337.831302] Lustre: Skipped 2 previous similar messages [ 2342.635134] Lustre: server umount lustre-MDT0000 complete [ 2342.889512] LustreError: 68709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.910686] LustreError: 68709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2347.218051] LustreError: 68689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643856 with bad export cookie 10720655045615226782 [ 2347.242578] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2347.243471] LustreError: 68689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2347.645684] Lustre: server umount lustre-MDT0001 complete [ 2360.079246] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 2363.184315] Key type lgssc unregistered [ 2363.402679] LNet: 73573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2363.409718] LNetError: 73573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.430706] LNet: Removed LNI 192.168.206.112@tcp [ 2364.334236] Key type .llcrypt unregistered [ 2364.336720] Key type ._llcrypt unregistered [ 2392.064995] Key type ._llcrypt registered [ 2392.067665] Key type .llcrypt registered [ 2392.196969] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2393.291697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2393.328910] alg: No test for adler32 (adler32-zlib) [ 2394.577749] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 2394.944474] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 2396.712649] Key type lgssc registered [ 2398.514315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2407.253962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2407.269482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.757881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.891865] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2418.144622] LustreError: 74983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.180261] LustreError: 74983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2423.264105] LustreError: 74984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.857402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2424.310849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2428.654719] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2437.249932] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2444.551678] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2451.334458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2451.764488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2453.814755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2459.275968] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2467.258851] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2476.661595] Lustre: Failing over lustre-MDT0000 [ 2477.244279] Lustre: server umount lustre-MDT0000 complete [ 2479.123317] LustreError: 74978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.591276] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.615090] Lustre: Skipped 2 previous similar messages [ 2484.201864] LustreError: 77153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.222754] LustreError: 77153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2487.183775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.323726] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2487.467689] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2487.622133] LustreError: 78072:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2487.631954] LustreError: 78072:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.112@tcp: cfg command failed: rc = -9 [ 2487.638796] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2487.638796] [ 2487.661307] LustreError: MGC192.168.206.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2487.684730] LustreError: 78060:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2487.690767] LustreError: 78060:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2487.716656] LustreError: 78060:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2487.723757] LustreError: 78060:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2487.760633] Lustre: server umount lustre-MDT0000 complete [ 2487.772391] LustreError: 78060:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2489.327249] LustreError: 74980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.363645] LustreError: 74980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2495.213650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.369567] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2495.379089] LustreError: 74975:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2495.685299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2495.749550] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2497.518686] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2500.100298] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2501.103394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2501.124977] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2501.163819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2510.304454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2510.311156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.323312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2511.334093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2511.334354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2513.505457] Lustre: server umount lustre-OST0000 complete [ 2521.575884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2521.587525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2521.606934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2521.615917] Lustre: Skipped 2 previous similar messages [ 2526.691401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2526.711706] Lustre: Skipped 1 previous similar message [ 2531.812659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2531.822151] Lustre: Skipped 1 previous similar message [ 2532.319959] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2532.532376] Lustre: server umount lustre-MDT0000 complete [ 2536.643249] LustreError: 74964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644045 with bad export cookie 13837926737947424509 [ 2536.663438] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2537.139726] Lustre: server umount lustre-MDT0001 complete [ 2549.631875] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 2552.439964] Key type lgssc unregistered [ 2552.729195] LNet: 79854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2552.741863] LNetError: 79854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2552.779296] LNet: Removed LNI 192.168.206.112@tcp [ 2553.615443] Key type .llcrypt unregistered [ 2553.619361] Key type ._llcrypt unregistered [ 2568.387380] Key type ._llcrypt registered [ 2568.390133] Key type .llcrypt registered [ 2568.543647] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 2568.859499] Key type .llcrypt unregistered [ 2568.861282] Key type ._llcrypt unregistered [ 2591.995187] Key type ._llcrypt registered [ 2591.998644] Key type .llcrypt registered [ 2592.132218] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2593.312152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2593.354185] alg: No test for adler32 (adler32-zlib) [ 2594.485841] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 2594.697291] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 2596.417133] Key type lgssc registered [ 2597.627616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2605.849818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2605.875581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2607.416360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2612.746623] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2617.835394] LustreError: 81747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.863713] LustreError: 81747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2622.666565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.048135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2627.844274] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2636.889354] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2643.236798] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2650.451205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2650.960603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2653.085613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2658.238713] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2667.519693] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2678.752376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2678.764489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.778977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2682.853535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2682.855188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2682.861420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2687.979984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2687.991587] Lustre: Skipped 2 previous similar messages [ 2692.063116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2692.297330] Lustre: server umount lustre-OST0000 complete [ 2698.208720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.228552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2698.234645] Lustre: Skipped 1 previous similar message [ 2701.888515] Lustre: server umount lustre-MDT0000 complete [ 2703.332270] LustreError: 81744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.348717] LustreError: 81744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2705.470099] LustreError: 81729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644214 with bad export cookie 9926105940774270924 [ 2705.474963] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.479416] LustreError: 81729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2705.815066] Lustre: server umount lustre-MDT0001 complete [ 2714.755758] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 2717.149060] Key type lgssc unregistered [ 2717.419114] LNet: 85282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2717.425746] LNetError: 85282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2717.447741] LNet: Removed LNI 192.168.206.112@tcp [ 2718.179903] Key type .llcrypt unregistered [ 2718.182462] Key type ._llcrypt unregistered [ 2745.135172] Key type ._llcrypt registered [ 2745.136871] Key type .llcrypt registered [ 2745.232448] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2746.337987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2746.529069] alg: No test for adler32 (adler32-zlib) [ 2747.759274] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 2748.035632] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 2749.800385] Key type lgssc registered [ 2751.047210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2762.998263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2763.047801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2764.871653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2771.067091] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2775.021150] LustreError: 86696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.047445] LustreError: 86696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2780.136901] LustreError: 86695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.066813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.444721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2787.791742] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2796.448193] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2803.074525] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2811.366093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2811.953534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.680507] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2830.195258] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2834.496514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2842.592423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2842.605410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2842.625263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.640694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2844.653907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.658387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.886335] Lustre: server umount lustre-OST0000 complete [ 2854.371465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2854.387451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.405618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.409732] Lustre: Skipped 1 previous similar message [ 2857.396382] Lustre: server umount lustre-MDT0000 complete [ 2860.000510] LustreError: 86695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2860.017047] LustreError: 86695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2861.417319] LustreError: 86677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644370 with bad export cookie 8847809427101870077 [ 2861.420759] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2861.424154] LustreError: 86677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2861.824172] Lustre: server umount lustre-MDT0001 complete [ 2872.655664] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 2874.907039] Key type lgssc unregistered [ 2875.167704] LNet: 90229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2875.177724] LNetError: 90229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2875.193841] LNet: Removed LNI 192.168.206.112@tcp [ 2875.924728] Key type .llcrypt unregistered [ 2875.926844] Key type ._llcrypt unregistered [ 2901.390581] Key type ._llcrypt registered [ 2901.404742] Key type .llcrypt registered [ 2901.567727] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2903.209636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2903.292860] alg: No test for adler32 (adler32-zlib) [ 2904.430690] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 2904.725720] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 2906.528716] Key type lgssc registered [ 2907.939274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2917.619840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2917.641683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2919.346878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.129040] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2929.634506] LustreError: 91640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2929.654747] LustreError: 91640:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2932.831074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.345958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2937.847151] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2944.994326] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2950.381258] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2957.132392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2957.546649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2963.232320] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2964.719656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2970.168238] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2980.324996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.335439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2980.336213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2980.346135] Lustre: Skipped 1 previous similar message [ 2985.442850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2985.454296] Lustre: Skipped 1 previous similar message [ 2986.441097] Lustre: server umount lustre-OST0000 complete [ 2990.566084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2990.582874] Lustre: Skipped 1 previous similar message [ 2990.590426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2990.605567] Lustre: Skipped 1 previous similar message [ 2995.685245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2995.693682] Lustre: Skipped 1 previous similar message [ 3000.802122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3000.815352] Lustre: Skipped 1 previous similar message [ 3004.385518] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3004.694178] Lustre: server umount lustre-MDT0000 complete [ 3005.921805] LustreError: 91641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3005.946337] LustreError: 91641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3007.876380] LustreError: 91619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644516 with bad export cookie 16348110921059078483 [ 3007.892115] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3007.898825] LustreError: 91619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3008.274796] Lustre: server umount lustre-MDT0001 complete [ 3020.073213] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 3023.649777] Key type lgssc unregistered [ 3024.040592] LNet: 95173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3024.056975] LNetError: 95173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3024.103500] LNet: Removed LNI 192.168.206.112@tcp [ 3025.375674] Key type .llcrypt unregistered [ 3025.377437] Key type ._llcrypt unregistered [ 3047.909241] Key type ._llcrypt registered [ 3047.911519] Key type .llcrypt registered [ 3048.045730] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 3048.859861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3048.906496] alg: No test for adler32 (adler32-zlib) [ 3049.897939] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 3050.100491] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 3051.800586] Key type lgssc registered [ 3052.846825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3058.865785] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3058.879281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3074.400150] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3079.519823] LustreError: 96536:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.112@tcp: failed processing log, type 4: rc = -110 [ 3106.143630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3111.913656] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3118.292593] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3123.391370] Lustre: server umount lustre-OST0000 complete [ 3131.989297] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 17:43:58 (1788644638) [ 3138.412457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3138.872460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3142.522794] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3144.164625] LustreError: 98176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3144.184490] LustreError: 98176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3148.258557] LustreError: 98177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.431421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.737067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3153.213689] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3159.131993] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3163.298580] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3168.935227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3169.383447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3175.058371] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3180.911220] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3184.761062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3195.367919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3195.368226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.373452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3195.379048] LustreError: Skipped 1 previous similar message [ 3195.411669] Lustre: Skipped 1 previous similar message [ 3200.485062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3200.493075] Lustre: Skipped 2 previous similar messages [ 3201.417029] Lustre: server umount lustre-OST0000 complete [ 3205.602241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.605302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3205.647419] Lustre: Skipped 1 previous similar message [ 3210.728825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3210.737890] Lustre: Skipped 2 previous similar messages [ 3211.221934] Lustre: server umount lustre-MDT0000 complete [ 3215.845829] LustreError: 100297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3215.853473] LustreError: 98157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644724 with bad export cookie 8177611092895179909 [ 3215.873074] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3215.886422] LustreError: 100297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3215.928803] LustreError: 98157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3216.393891] Lustre: server umount lustre-MDT0001 complete [ 3227.597620] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 3230.144907] Key type lgssc unregistered [ 3230.530704] LNet: 101865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3230.543824] LNetError: 101865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3230.560655] LNet: Removed LNI 192.168.206.112@tcp [ 3231.305232] Key type .llcrypt unregistered [ 3231.307425] Key type ._llcrypt unregistered [ 3267.206957] Key type ._llcrypt registered [ 3267.209472] Key type .llcrypt registered [ 3267.291908] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 3268.034738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3268.090386] alg: No test for adler32 (adler32-zlib) [ 3269.114451] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 3269.366123] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 3271.031113] Key type lgssc registered [ 3271.998709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3278.902177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3278.924150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3280.382536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3284.695524] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3290.597780] LustreError: 104025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3290.618560] LustreError: 104025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3293.514348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.936468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3298.668375] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3305.603201] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3311.071727] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3318.021351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3318.378920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3324.618477] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3325.563895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3331.976378] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3345.378874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3345.385487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.399924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3346.401672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3346.405773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3346.419261] Lustre: Skipped 1 previous similar message [ 3347.645353] Lustre: server umount lustre-OST0000 complete [ 3351.523983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3351.526413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3351.541149] Lustre: Skipped 1 previous similar message [ 3351.556407] Lustre: Skipped 1 previous similar message [ 3356.646225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3356.654563] Lustre: Skipped 1 previous similar message [ 3357.042715] Lustre: server umount lustre-MDT0000 complete [ 3360.353582] LustreError: 104006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644869 with bad export cookie 7064178392723011004 [ 3360.365346] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3360.375718] LustreError: 104006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3360.713548] Lustre: server umount lustre-MDT0001 complete [ 3370.527917] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 3373.103227] Key type lgssc unregistered [ 3373.387508] LNet: 107555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3373.397371] LNetError: 107555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3373.415793] LNet: Removed LNI 192.168.206.112@tcp [ 3374.259321] Key type .llcrypt unregistered [ 3374.264399] Key type ._llcrypt unregistered [ 3391.313257] Key type ._llcrypt registered [ 3391.315588] Key type .llcrypt registered [ 3391.407372] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 3392.484441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3392.546327] alg: No test for adler32 (adler32-zlib) [ 3393.553445] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 3393.816991] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 3395.495165] Key type lgssc registered [ 3396.760479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3403.582814] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3403.614868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3419.233223] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3424.351458] LustreError: 108798:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.112@tcp: failed processing log, type 4: rc = -110 [ 3451.040560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3457.449542] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3465.490481] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3471.400650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3471.698858] LustreError: MGC192.168.206.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3471.706958] LustreError: 109823:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3471.712808] LustreError: 109823:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3471.721893] LustreError: 109823:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3471.808356] Lustre: server umount lustre-MDT0000 complete [ 3471.811483] LustreError: 109823:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3492.319224] Lustre: 108135:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788644985/real 1788644985] req@0000000070595911 x1875530116762880/t0(0) o400->MGC192.168.206.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788645001 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3492.379426] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3492.394847] LustreError: 108810:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3492.548297] Lustre: server umount lustre-OST0000 complete [ 3504.216821] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 3516.856678] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 3552.490520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3552.710963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3552.738446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3552.798338] Lustre: lustre-MDT0000: new disk, initializing [ 3552.846126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3552.861629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3557.316222] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3567.119539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3567.215624] Lustre: 112615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3567.257917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3567.261826] Lustre: Skipped 1 previous similar message [ 3567.342205] Lustre: lustre-MDT0001: new disk, initializing [ 3567.437027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3567.488822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3567.500030] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3571.490763] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3579.828706] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3585.174984] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3590.097955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3590.287106] Lustre: lustre-OST0000: new disk, initializing [ 3590.289944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3590.294959] Lustre: 114580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3590.372902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3595.009474] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3596.826206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3596.840023] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3596.929926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3602.732436] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3606.134992] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3606.297373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3609.855133] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3610.043971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3612.131390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3612.131685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3612.139664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3612.151083] Lustre: Skipped 1 previous similar message [ 3617.253127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3617.258409] Lustre: Skipped 2 previous similar messages [ 3622.371552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3622.380995] Lustre: Skipped 1 previous similar message [ 3625.951108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3626.050474] Lustre: server umount lustre-OST0000 complete [ 3632.607912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3632.613323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3632.627659] Lustre: Skipped 1 previous similar message [ 3635.989357] Lustre: server umount lustre-MDT0000 complete [ 3638.814211] LustreError: 112607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645147 with bad export cookie 2101407509397758215 [ 3638.816299] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3638.820036] LustreError: 112607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3639.126457] Lustre: server umount lustre-MDT0001 complete [ 3646.176506] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 17:52:32 (1788645152) [ 3648.852590] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3661.501990] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 3668.976449] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 3707.470059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.741847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3707.783222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3707.885309] Lustre: lustre-MDT0000: new disk, initializing [ 3707.949755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3707.966168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3712.799889] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3723.703187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3723.871424] Lustre: 119487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3723.939102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3723.943035] Lustre: Skipped 1 previous similar message [ 3724.008696] Lustre: lustre-MDT0001: new disk, initializing [ 3724.149788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3724.307969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3724.333294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3728.488814] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3738.733543] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3745.860506] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3753.277309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3753.610841] Lustre: lustre-OST0000: new disk, initializing [ 3753.614866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3753.625809] Lustre: 121456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3753.715854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3759.172047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3759.193953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3759.311504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3760.314798] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3769.559860] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3773.894665] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3774.037398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3778.379482] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3778.555881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3785.200641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3785.201313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3785.203412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3785.203419] Lustre: Skipped 1 previous similar message [ 3785.213952] LustreError: Skipped 1 previous similar message [ 3785.222625] Lustre: Skipped 1 previous similar message [ 3786.487294] Lustre: server umount lustre-OST0000 complete [ 3790.316419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3790.334056] Lustre: Skipped 1 previous similar message [ 3795.425529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3795.435865] Lustre: Skipped 4 previous similar messages [ 3804.130579] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3804.460577] Lustre: server umount lustre-MDT0000 complete [ 3805.665779] LustreError: 119499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3805.685652] LustreError: 119499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3807.833424] LustreError: 119478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645316 with bad export cookie 2101407509397759097 [ 3807.836116] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3807.848521] LustreError: 119478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3808.127660] Lustre: server umount lustre-MDT0001 complete [ 3816.504198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3816.882353] LustreError: 123674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3816.964505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3821.599276] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3822.049066] LustreError: 123675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3826.159253] LustreError: 123674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3830.532900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3834.660035] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3841.277587] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3847.135963] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3854.152520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3860.465764] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3868.323992] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3874.763943] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3881.443743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3881.451983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3881.467430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3881.473783] Lustre: Skipped 2 previous similar messages [ 3884.809825] Lustre: server umount lustre-OST0000 complete [ 3892.193377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3892.200273] LustreError: Skipped 1 previous similar message [ 3892.206822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3892.216038] Lustre: Skipped 1 previous similar message [ 3894.317632] Lustre: server umount lustre-MDT0000 complete [ 3897.832387] LustreError: 124927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3897.855538] LustreError: 124927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3897.917402] LustreError: 123654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645406 with bad export cookie 2101407509397759979 [ 3897.923751] LustreError: 123654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3897.924667] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3898.287972] Lustre: server umount lustre-MDT0001 complete [ 3908.983187] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 3911.890266] Key type lgssc unregistered [ 3912.209679] LNet: 127301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3912.213570] LNetError: 127301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3912.227644] LNet: Removed LNI 192.168.206.112@tcp [ 3912.943601] Key type .llcrypt unregistered [ 3912.945751] Key type ._llcrypt unregistered [ 3930.418929] Key type ._llcrypt registered [ 3930.421827] Key type .llcrypt registered [ 3930.540526] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 3945.723401] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 3946.618502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3946.708312] alg: No test for adler32 (adler32-zlib) [ 3948.027604] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 3948.383329] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 3950.167607] Key type lgssc registered [ 3951.739435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3994.509800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3994.529989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3995.732619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3995.761783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3995.840357] Lustre: lustre-MDT0000: new disk, initializing [ 3995.942510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.970791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4001.510850] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4012.489862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.613726] Lustre: 130451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4012.658513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4012.662396] Lustre: Skipped 1 previous similar message [ 4012.724839] Lustre: lustre-MDT0001: new disk, initializing [ 4012.795234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4012.849000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4012.864500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4017.743163] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4028.746131] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4034.544923] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4041.024808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4041.294671] Lustre: lustre-OST0000: new disk, initializing [ 4041.300892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4041.307845] Lustre: 132424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4041.396731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4047.813566] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4050.486449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4050.511262] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4050.655958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4056.231770] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4060.772025] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4060.984516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4065.399459] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4065.659771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4071.394946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4071.398626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4071.402454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4071.420423] Lustre: Skipped 1 previous similar message [ 4073.845343] Lustre: server umount lustre-OST0000 complete [ 4079.585969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4079.592467] LustreError: Skipped 1 previous similar message [ 4079.595275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.602290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4079.605563] Lustre: Skipped 1 previous similar message [ 4081.633461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4081.641736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4081.670407] Lustre: Skipped 1 previous similar message [ 4083.890677] Lustre: server umount lustre-MDT0000 complete [ 4088.799318] LustreError: 130442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645597 with bad export cookie 12333381841342185207 [ 4088.809498] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4089.093705] Lustre: server umount lustre-MDT0001 complete [ 4100.182813] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:00:06 (1788645606) [ 4107.325432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4108.062921] LustreError: 134834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4108.078111] LustreError: 134834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4108.142896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4113.048320] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4113.394162] LustreError: 134835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4117.479939] LustreError: 134834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4121.125763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4121.611218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4126.158864] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4134.189330] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4140.361587] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4143.083118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.097883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4143.103342] Lustre: Skipped 1 previous similar message [ 4143.119896] Lustre: Skipped 1 previous similar message [ 4149.078781] Lustre: server umount lustre-MDT0000 complete [ 4151.267088] LustreError: 134830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4151.280085] LustreError: 134830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4153.866453] LustreError: 134815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645662 with bad export cookie 12333381841342186089 [ 4153.868270] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4153.885271] LustreError: 134815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4154.699677] Lustre: server umount lustre-MDT0001 complete [ 4165.356201] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:01:11 (1788645671) [ 4172.248301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4187.871572] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4192.994212] LustreError: 137261:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.112@tcp: failed processing log, type 4: rc = -110 [ 4218.719614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4226.939066] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4235.701717] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4238.508898] Lustre: server umount lustre-OST0000 complete [ 4245.998664] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:02:32 (1788645752) [ 4253.063408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4253.477954] LustreError: 138707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4253.574384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4257.036529] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4263.909040] LustreError: 138708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4263.933660] LustreError: 138708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4265.643193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4270.797925] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4278.683569] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4285.909257] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4293.331971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4293.866258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4293.876461] Lustre: Skipped 1 previous similar message [ 4300.803251] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4309.044406] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4317.168095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4317.176619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4317.189901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4317.200510] Lustre: Skipped 2 previous similar messages [ 4322.812975] Lustre: server umount lustre-MDT0000 complete [ 4325.860800] LustreError: 138702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.875053] LustreError: 138702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4326.798137] LustreError: 138687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645835 with bad export cookie 12333381841342186754 [ 4326.799306] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4326.817583] LustreError: 138687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4327.106640] Lustre: server umount lustre-MDT0001 complete [ 4341.526069] Lustre: server umount lustre-OST0000 complete [ 4350.960560] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:04:17 (1788645857) [ 4358.388771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.283981] LustreError: 142361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.364443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4363.978343] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4372.723986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.048765] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4384.133507] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4391.596851] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4400.903412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4404.476745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4408.582751] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4415.965101] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4421.242277] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4421.598121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4428.520349] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4429.020615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4433.378489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.381119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4433.410265] Lustre: Skipped 3 previous similar messages [ 4433.437620] Lustre: Skipped 4 previous similar messages [ 4438.477723] Lustre: server umount lustre-OST0000 complete [ 4449.660894] Lustre: server umount lustre-MDT0000 complete [ 4453.595136] LustreError: 142342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645962 with bad export cookie 12333381841342187825 [ 4453.622899] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4453.627992] LustreError: 142342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4454.023236] Lustre: server umount lustre-MDT0001 complete [ 4464.827342] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:06:10 (1788645970) [ 4471.740195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4487.455560] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4492.576527] LustreError: 146522:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.112@tcp: failed processing log, type 4: rc = -110 [ 4518.367415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4518.377428] Lustre: Skipped 2 previous similar messages [ 4524.597843] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4531.703038] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4537.492677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4537.822856] LustreError: 146547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4537.850314] LustreError: 146547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4537.977085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4542.338278] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4551.084783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4556.527241] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4563.989630] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4570.069970] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4574.509538] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4574.667595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4579.084970] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4579.308292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4582.368663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4582.378987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.390286] Lustre: Skipped 2 previous similar messages [ 4582.399990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4582.407506] Lustre: Skipped 4 previous similar messages [ 4587.452682] Lustre: server umount lustre-MDT0000 complete [ 4590.835463] LustreError: 146528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646099 with bad export cookie 12333381841342188847 [ 4590.852341] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4591.155379] Lustre: server umount lustre-MDT0001 complete [ 4604.848259] Lustre: server umount lustre-OST0000 complete [ 4612.496769] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:08:38 (1788646118) [ 4619.337604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4634.975872] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4640.095457] LustreError: 150713:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.112@tcp: failed processing log, type 4: rc = -110 [ 4673.202888] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4681.091968] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4687.444396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4688.087436] LustreError: 150738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4688.103524] LustreError: 150738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4688.216041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4692.762139] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4701.400711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4707.709641] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4717.654217] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4724.504744] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4732.208547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4732.423468] Lustre: lustre-OST0001: new disk, initializing [ 4732.428748] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4732.440267] Lustre: 153589:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4738.174280] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4739.649536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4739.662101] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4739.753159] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4747.279237] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4753.402907] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4753.809547] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4759.483302] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4759.775765] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4763.619933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4763.641551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4763.665126] Lustre: Skipped 2 previous similar messages [ 4763.672703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4763.679770] Lustre: Skipped 3 previous similar messages [ 4769.248177] Lustre: server umount lustre-OST0000 complete [ 4781.026034] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4781.051054] LustreError: Skipped 2 previous similar messages [ 4782.641906] Lustre: server umount lustre-OST0001 complete [ 4794.775546] Lustre: server umount lustre-MDT0000 complete [ 4799.413278] LustreError: 150720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646308 with bad export cookie 12333381841342189799 [ 4799.416807] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4799.421860] LustreError: 150720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4799.829898] Lustre: server umount lustre-MDT0001 complete [ 4828.384249] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:12:13 (1788646333) [ 4830.480230] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4832.344561] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4834.759724] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:12:20 (1788646340) [ 4844.561354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4845.107125] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4845.235853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4845.769517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4845.781165] Lustre: Skipped 6 previous similar messages [ 4851.970024] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4860.831897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.063826] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4861.094958] Lustre: 157151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4865.582136] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4872.938955] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4878.526269] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4884.666269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4884.881553] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4890.752887] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4892.897156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4898.808489] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4904.226985] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4904.519755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4910.023597] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4910.535334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4913.635180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4913.636548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4913.660514] Lustre: Skipped 5 previous similar messages [ 4913.663917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4913.663925] Lustre: Skipped 7 previous similar messages [ 4913.714755] LustreError: Skipped 1 previous similar message [ 4919.402812] Lustre: server umount lustre-OST0000 complete [ 4943.510830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4952.399579] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4963.194037] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4968.524241] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4968.571912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4968.572074] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4971.662290] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4971.965127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4978.052241] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4978.327708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4983.720381] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c55d99d7000.ost_server_uuid 50 [ 4985.463586] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8c55d99d7000.ost_server_uuid in FULL state after 0 sec [ 4985.753288] LustreError: 158209:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.12@tcp) returned error from blocking AST (req@00000000a48d7e21 x1875530698989696 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000528a99c9/0xab28f9708c745a41 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.12@tcp remote: 0x8ab19bfe55aaae02 expref: 5 pid: 157159 timeout: 5085 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4985.783534] LustreError: lustre-MDT0000: A client on nid 192.168.206.12@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4985.791906] LustreError: 157148:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.12@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000528a99c9/0xab28f9708c745a41 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.12@tcp remote: 0x8ab19bfe55aaae02 expref: 6 pid: 157159 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4991.952142] Lustre: server umount lustre-OST0000 complete [ 5005.439156] Lustre: server umount lustre-MDT0000 complete [ 5009.069133] LustreError: 157144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646517 with bad export cookie 12333381841342190975 [ 5009.072205] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5009.082782] LustreError: 157144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5018.758470] Lustre: server umount lustre-MDT0001 complete [ 5029.897134] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 5032.574841] Key type lgssc unregistered [ 5032.919696] LNet: 163267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5032.939511] LNetError: 163267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5033.961587] LNet: Removed LNI 192.168.206.112@tcp [ 5034.838494] Key type .llcrypt unregistered [ 5034.842418] Key type ._llcrypt unregistered [ 5057.314976] Key type ._llcrypt registered [ 5057.318651] Key type .llcrypt registered [ 5057.437619] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 5059.122142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5059.153185] alg: No test for adler32 (adler32-zlib) [ 5060.197754] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 5060.512226] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 5062.199157] Key type lgssc registered [ 5063.376914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5072.003972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5072.028341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5073.420319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5079.266921] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5083.623770] LustreError: 164673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5083.637793] LustreError: 164673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5088.147040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5088.625274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5093.704149] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5101.706438] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5107.434680] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5114.332351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5114.703037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5118.788195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5120.787063] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5128.189292] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5137.646699] Lustre: Failing over lustre-MDT0000 [ 5137.890868] Lustre: server umount lustre-MDT0000 complete [ 5139.447742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.461518] LustreError: 164667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5139.475286] Lustre: Skipped 2 previous similar messages [ 5139.488578] LustreError: 164667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5144.548701] LustreError: 165402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5144.578779] LustreError: 165402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5149.667724] LustreError: 164667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5149.698518] LustreError: 164667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5150.722074] LustreError: 164654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.12@tcp arrived at 1788646659 with bad export cookie 1990853434718635888 [ 5150.736137] LustreError: 164654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5154.785667] LustreError: 164672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5154.795777] LustreError: 164672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5155.809133] Lustre: 163980:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788646648/real 1788646648] req@00000000b45c5b4b x1875531864761600/t0(0) o400->MGC192.168.206.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788646664 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5155.838942] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5157.756462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.048261] LustreError: 164673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5166.059130] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1ba0eea6a04547d9 [ 5166.069399] LustreError: 164673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5166.075969] Lustre: MGC192.168.206.112@tcp: Connection restored to 0@lo (at 0@lo) [ 5166.263145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.330266] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5170.058963] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5171.681733] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5171.692313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5172.254164] Lustre: lustre-MDT0000: Denying connection for new client 08d95916-4463-45b3-a696-2e5126ee0302 (at 192.168.206.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5181.941634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5181.947843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5181.949742] Lustre: Skipped 1 previous similar message [ 5181.967082] Lustre: Skipped 1 previous similar message [ 5186.679582] Lustre: server umount lustre-OST0000 complete [ 5190.190614] LustreError: 168408:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5190.200587] Lustre: 167756:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5190.208711] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5190.220359] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5190.226490] LustreError: 167756:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bd22a76d x1875531864781056/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5190.234841] LustreError: 167756:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5190.239304] LustreError: 167756:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5190.247439] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 5190.493381] Lustre: server umount lustre-MDT0000 complete [ 5192.161897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5192.184765] LustreError: 164669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5194.193240] LustreError: 164652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646703 with bad export cookie 1990853434718635993 [ 5194.194678] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5194.209142] LustreError: 164652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5196.597891] Lustre: server umount lustre-MDT0001 complete [ 5206.260663] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 5208.862655] Key type lgssc unregistered [ 5209.107842] LNet: 169158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5209.112836] LNetError: 169158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5209.146335] LNet: Removed LNI 192.168.206.112@tcp [ 5209.996250] Key type .llcrypt unregistered [ 5209.999932] Key type ._llcrypt unregistered [ 5228.180724] Key type ._llcrypt registered [ 5228.184417] Key type .llcrypt registered [ 5228.275672] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 5228.946364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5228.975161] alg: No test for adler32 (adler32-zlib) [ 5230.021662] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 5230.231755] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 5231.871269] Key type lgssc registered [ 5232.783313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5239.512304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5239.534644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5240.895138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5244.519880] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5251.041940] LustreError: 170545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5251.072183] LustreError: 170545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5251.265630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5251.529077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5255.162244] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5261.362340] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5266.358217] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5272.436613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5272.711818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5277.180869] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5282.864122] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5286.069599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5292.512658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5292.521164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5292.537283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5296.612334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5296.617491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5296.619929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5296.631562] Lustre: Skipped 1 previous similar message [ 5297.830994] Lustre: server umount lustre-OST0000 complete [ 5301.729946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5301.742876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5301.749757] Lustre: Skipped 1 previous similar message [ 5306.850131] LustreError: 170545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5306.858720] LustreError: 170545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5306.883920] Lustre: server umount lustre-MDT0000 complete [ 5309.764908] LustreError: 170527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646818 with bad export cookie 4417982959911130790 [ 5309.770259] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5309.775793] LustreError: 170527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5310.011517] Lustre: server umount lustre-MDT0001 complete [ 5318.370342] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 5320.374427] Key type lgssc unregistered [ 5320.576411] LNet: 174075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5320.583612] LNetError: 174075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5320.595263] LNet: Removed LNI 192.168.206.112@tcp [ 5321.140909] Key type .llcrypt unregistered [ 5321.143184] Key type ._llcrypt unregistered [ 5347.837291] Key type ._llcrypt registered [ 5347.840739] Key type .llcrypt registered [ 5347.933529] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 5348.619747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5348.635533] alg: No test for adler32 (adler32-zlib) [ 5349.672940] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 5349.919678] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 5351.567177] Key type lgssc registered [ 5352.424079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5358.960530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5358.981198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5360.399257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5363.681929] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5370.865507] LustreError: 175754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5370.897605] LustreError: 175754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5371.797320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5372.271301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5377.510718] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5386.147126] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5391.657779] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5398.876260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5399.240425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5404.629480] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5405.372671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5411.935537] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5426.147253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5426.154161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5426.159799] LustreError: Skipped 1 previous similar message [ 5426.209109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5428.203669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5428.207326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5428.211063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5428.248335] Lustre: Skipped 1 previous similar message [ 5431.313097] Lustre: server umount lustre-OST0000 complete [ 5438.435484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.465599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5438.470759] Lustre: Skipped 1 previous similar message [ 5441.340616] Lustre: server umount lustre-MDT0000 complete [ 5443.556161] LustreError: 175748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5443.571696] LustreError: 175748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5444.672118] LustreError: 175734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646953 with bad export cookie 4348845151121283106 [ 5444.687114] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5444.940285] Lustre: server umount lustre-MDT0001 complete [ 5455.150995] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 5457.688144] Key type lgssc unregistered [ 5457.911040] LNet: 179311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5457.916753] LNetError: 179311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5457.932849] LNet: Removed LNI 192.168.206.112@tcp [ 5458.666995] Key type .llcrypt unregistered [ 5458.670517] Key type ._llcrypt unregistered [ 5479.264614] Key type ._llcrypt registered [ 5479.266562] Key type .llcrypt registered [ 5479.354408] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 5480.526553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5480.585218] alg: No test for adler32 (adler32-zlib) [ 5481.649614] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 5481.885337] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 5483.591183] Key type lgssc registered [ 5484.679763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5495.101256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5495.132805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5496.459552] Lustre: *** cfs_fail_loc=135, val=0*** [ 5496.510981] LustreError: 180829:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5496.521717] LustreError: 180829:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.112@tcp: cfg command failed: rc = -2 [ 5496.527324] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5496.527324] [ 5496.539478] LustreError: MGC192.168.206.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5496.556313] LustreError: 180790:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5496.562788] LustreError: 180790:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 5496.566960] LustreError: 180790:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5496.571444] LustreError: 180790:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5496.627924] Lustre: server umount lustre-MDT0000 complete [ 5496.630623] LustreError: 180790:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5501.949752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.364523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5506.370353] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5512.675083] LustreError: 181200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5512.694105] LustreError: 181200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5514.789342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.193349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5519.767716] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5527.296948] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5531.963253] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5535.712762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5535.718909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5535.732141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5537.761427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5537.761450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5537.778653] Lustre: Skipped 1 previous similar message [ 5539.810149] LustreError: 181200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5539.828742] LustreError: 181200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5539.905119] Lustre: server umount lustre-MDT0000 complete [ 5543.411929] LustreError: 181182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647052 with bad export cookie 9890459976583736004 [ 5543.430410] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5543.431246] LustreError: 181182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5543.730942] Lustre: server umount lustre-MDT0001 complete [ 5555.444363] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 5558.132293] Key type lgssc unregistered [ 5558.474102] LNet: 183520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5558.482825] LNetError: 183520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5558.506673] LNet: Removed LNI 192.168.206.112@tcp [ 5559.353530] Key type .llcrypt unregistered [ 5559.357947] Key type ._llcrypt unregistered [ 5581.792480] Key type ._llcrypt registered [ 5581.795690] Key type .llcrypt registered [ 5581.888682] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 5582.204642] Key type .llcrypt unregistered [ 5582.209993] Key type ._llcrypt unregistered [ 5599.582868] Key type ._llcrypt registered [ 5599.585955] Key type .llcrypt registered [ 5599.684799] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 5600.383903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5600.408679] alg: No test for adler32 (adler32-zlib) [ 5601.416518] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 5601.803537] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 5603.463150] Key type lgssc registered [ 5604.675781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5612.527227] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5612.576415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5628.319767] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5633.439966] LustreError: 185647:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.112@tcp: failed processing log, type 4: rc = -110 [ 5660.127636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5665.481378] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5673.626895] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5679.689244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5680.052676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5680.105088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5683.529134] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5685.221681] LustreError: 185673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5685.236222] LustreError: 185673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5690.338680] LustreError: 185672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5690.721246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.116130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5695.129347] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5701.200608] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5706.605465] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5709.292234] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5720.036296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5720.041101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5720.047925] Lustre: Skipped 1 previous similar message [ 5725.167725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5725.177118] Lustre: Skipped 2 previous similar messages [ 5730.290085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5730.310352] Lustre: Skipped 1 previous similar message [ 5733.343131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5733.547341] Lustre: server umount lustre-OST0000 complete [ 5737.448672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5737.463143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.477547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5740.513319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.710164] Lustre: server umount lustre-MDT0000 complete [ 5745.635257] LustreError: 186690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.651566] LustreError: 186690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5747.261260] LustreError: 185653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647256 with bad export cookie 9429673331409012769 [ 5747.270413] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5747.639331] Lustre: server umount lustre-MDT0001 complete [ 5757.351863] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 5759.880749] Key type lgssc unregistered [ 5760.173847] LNet: 189614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5760.183403] LNetError: 189614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5761.255573] LNet: Removed LNI 192.168.206.112@tcp [ 5762.037970] Key type .llcrypt unregistered [ 5762.041396] Key type ._llcrypt unregistered [ 5782.216450] Key type ._llcrypt registered [ 5782.219696] Key type .llcrypt registered [ 5782.328532] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 5783.772322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5783.802615] alg: No test for adler32 (adler32-zlib) [ 5784.911362] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 5785.204938] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 5786.879162] Key type lgssc registered [ 5788.283968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5797.143261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5797.184412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5798.734362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5803.533832] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5809.129443] LustreError: 191001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5809.147375] LustreError: 191001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5812.480684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5813.124887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5818.254361] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5825.871393] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5830.867383] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5836.846881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5837.231941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.087229] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5844.415242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5850.544316] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5855.843662] Lustre: Failing over lustre-MDT0000 [ 5856.265932] Lustre: server umount lustre-MDT0000 complete [ 5857.809318] LustreError: 190997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5859.296421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5859.315562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.334411] LustreError: 191002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5859.817236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.835627] Lustre: Skipped 1 previous similar message [ 5862.891524] LustreError: 191987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5862.913020] LustreError: 191987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5868.006292] LustreError: 190996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.024383] LustreError: 190996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5875.170928] Lustre: 190312:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788647368/real 1788647368] req@0000000023d532fe x1875532624980736/t0(0) o400->MGC192.168.206.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788647384 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5875.195591] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5875.827633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.248293] LustreError: 191987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5878.265813] LustreError: 191987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5885.419605] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2d87ebfbb78bb027 [ 5885.429547] Lustre: MGC192.168.206.112@tcp: Connection restored to 0@lo (at 0@lo) [ 5885.693418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5885.753459] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5888.488111] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5890.032969] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5891.048819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5891.085081] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5891.128643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5895.096309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5903.271683] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5909.990573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5910.004509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5910.032720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5911.545431] LustreError: 192824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5911.563198] LustreError: 192824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5911.879471] Lustre: server umount lustre-OST0000 complete [ 5916.640344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5916.640871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.654925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5916.680295] Lustre: Skipped 2 previous similar messages [ 5921.762790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5921.776330] Lustre: Skipped 2 previous similar messages [ 5923.272422] Lustre: server umount lustre-MDT0000 complete [ 5927.654125] LustreError: 190983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647436 with bad export cookie 3280850319911399463 [ 5927.662044] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5927.672334] LustreError: 190983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5927.888102] Lustre: server umount lustre-MDT0001 complete [ 5938.710807] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 5943.218677] Key type lgssc unregistered [ 5943.598400] LNet: 195891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5943.606408] LNetError: 195891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5943.628532] LNet: Removed LNI 192.168.206.112@tcp [ 5944.563889] Key type .llcrypt unregistered [ 5944.566472] Key type ._llcrypt unregistered [ 5966.613784] Key type ._llcrypt registered [ 5966.616152] Key type .llcrypt registered [ 5966.716607] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 5967.804311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5967.830260] alg: No test for adler32 (adler32-zlib) [ 5968.876144] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 5969.122255] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 5970.807218] Key type lgssc registered [ 5972.095696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5981.755775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5981.775559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5983.325629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5988.360287] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5993.443059] LustreError: 197302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5993.481696] LustreError: 197302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5998.569444] LustreError: 197303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5999.381317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.828626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6003.399595] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6010.474606] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6015.967955] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6021.956725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6022.304391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6027.441302] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6028.476380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6034.225273] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6037.405952] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6041.883535] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6054.801585] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6064.614204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6064.617366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6064.624691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6064.637666] Lustre: Skipped 1 previous similar message [ 6064.646028] Lustre: Skipped 1 previous similar message [ 6066.851181] Lustre: server umount lustre-OST0000 complete [ 6071.776740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6071.781935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6071.798636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6074.853144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6074.864961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.574142] Lustre: server umount lustre-MDT0000 complete [ 6079.971059] LustreError: 197303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6080.000961] LustreError: 197303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6080.446344] LustreError: 197282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647589 with bad export cookie 2219308769272725113 [ 6080.456697] LustreError: 197282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6080.464147] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6080.777380] Lustre: server umount lustre-MDT0001 complete [ 6091.380991] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 6094.447803] Key type lgssc unregistered [ 6094.778473] LNet: 201143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6094.789400] LNetError: 201143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6094.803424] LNet: Removed LNI 192.168.206.112@tcp [ 6095.742859] Key type .llcrypt unregistered [ 6095.744667] Key type ._llcrypt unregistered [ 6113.102311] Key type ._llcrypt registered [ 6113.107852] Key type .llcrypt registered [ 6113.261381] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 6115.176815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6115.185411] alg: No test for adler32 (adler32-zlib) [ 6116.256908] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 6116.518063] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 6118.231202] Key type lgssc registered [ 6119.572087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6128.245902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6128.261989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6129.710313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6134.089603] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6139.875038] LustreError: 202413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6139.904773] LustreError: 202413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6143.093162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6143.613747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6148.335610] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6157.301113] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6163.320700] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6172.499042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6173.024176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6175.095380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6179.364772] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6187.186682] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6193.306573] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6204.165396] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6210.431627] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6217.926413] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6226.399994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6226.413179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.438284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6229.474728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6229.477207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6229.477627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6232.832596] Lustre: server umount lustre-OST0000 complete [ 6239.713680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6239.717744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6239.743014] Lustre: Skipped 1 previous similar message [ 6239.760964] Lustre: Skipped 2 previous similar messages [ 6242.896552] Lustre: server umount lustre-MDT0000 complete [ 6244.836531] LustreError: 202414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.860360] LustreError: 202414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6246.500989] LustreError: 202393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647755 with bad export cookie 9017262497266810524 [ 6246.506511] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6246.512219] LustreError: 202393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6247.064604] Lustre: server umount lustre-MDT0001 complete [ 6257.076717] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 6259.843348] Key type lgssc unregistered [ 6260.177980] LNet: 207288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6260.184717] LNetError: 207288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6260.207212] LNet: Removed LNI 192.168.206.112@tcp [ 6260.958981] Key type .llcrypt unregistered [ 6260.963494] Key type ._llcrypt unregistered [ 6283.939823] Key type ._llcrypt registered [ 6283.942505] Key type .llcrypt registered [ 6284.047790] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 6285.146909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6285.387518] alg: No test for adler32 (adler32-zlib) [ 6286.449321] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 6286.870166] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 6288.607991] Key type lgssc registered [ 6289.902247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6298.738788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6298.787437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6300.400639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6305.169631] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6305.765367] LustreError: 208699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.782490] LustreError: 208699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6310.883286] LustreError: 208698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6313.603571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6313.948607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6318.156401] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6325.209321] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6330.807482] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6338.292529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6338.572532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6345.646905] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6345.738276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6354.220547] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6369.251163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6369.259067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6369.274167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6371.301641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6371.304116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6371.320253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.992814] Lustre: server umount lustre-OST0000 complete [ 6380.514562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6380.530397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6380.564065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6380.570354] Lustre: Skipped 1 previous similar message [ 6385.715486] Lustre: server umount lustre-MDT0000 complete [ 6386.656800] LustreError: 208699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6386.711178] LustreError: 208699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6391.019744] LustreError: 208678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647899 with bad export cookie 1025223072833754545 [ 6391.041741] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6391.537871] Lustre: server umount lustre-MDT0001 complete [ 6404.076891] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 6407.864687] Key type lgssc unregistered [ 6408.100439] LNet: 212378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6408.114717] LNetError: 212378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6408.134245] LNet: Removed LNI 192.168.206.112@tcp [ 6409.104114] Key type .llcrypt unregistered [ 6409.111727] Key type ._llcrypt unregistered [ 6429.472508] Key type ._llcrypt registered [ 6429.475638] Key type .llcrypt registered [ 6429.630594] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 6431.078757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6431.108873] alg: No test for adler32 (adler32-zlib) [ 6432.398858] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 6432.782701] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 6434.591417] Key type lgssc registered [ 6436.098296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6445.973617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6446.054485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6447.730346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6453.924815] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6457.315080] LustreError: 213642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6457.345595] LustreError: 213642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6462.448266] LustreError: 213643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6464.603928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6465.257640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6469.753875] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6477.693674] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6483.757752] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6490.316790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6490.677664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6492.733098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6496.281197] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6503.080885] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6520.289573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6520.294222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6520.304549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6520.321592] Lustre: Skipped 1 previous similar message [ 6525.246745] Lustre: server umount lustre-OST0000 complete [ 6530.529224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6530.544695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6530.559691] Lustre: Skipped 2 previous similar messages [ 6534.966734] Lustre: server umount lustre-MDT0000 complete [ 6535.653922] LustreError: 213642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6535.665264] LustreError: 213642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6538.154977] LustreError: 213625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648047 with bad export cookie 11804147862653708100 [ 6538.159713] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6538.164984] LustreError: 213625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6538.404697] Lustre: server umount lustre-MDT0001 complete [ 6548.229220] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 6551.093719] Key type lgssc unregistered [ 6551.362032] LNet: 217320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6551.368323] LNetError: 217320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6551.385675] LNet: Removed LNI 192.168.206.112@tcp [ 6552.109195] Key type .llcrypt unregistered [ 6552.111189] Key type ._llcrypt unregistered [ 6571.998405] Key type ._llcrypt registered [ 6571.999606] Key type .llcrypt registered [ 6572.170421] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 6573.532980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6573.697355] alg: No test for adler32 (adler32-zlib) [ 6574.970983] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 6575.322919] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 6577.015160] Key type lgssc registered [ 6578.691871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6589.818688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6589.834561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6591.468439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6597.722104] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6601.712391] LustreError: 218586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6601.736297] LustreError: 218586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6606.817919] LustreError: 218587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6608.207895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6608.592083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6612.764346] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6619.950442] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6625.357867] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6631.834361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6632.451572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6636.531158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6640.493693] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6650.481509] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6660.417050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6660.493083] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6660.714549] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6667.456603] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6675.200602] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6676.614993] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6688.413917] Lustre: Permanently deactivating lustre-OST0001 [ 6688.434950] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6697.564206] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6707.105657] Lustre: Permanently reactivating lustre-OST0001 [ 6707.128365] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6707.135655] Lustre: Skipped 2 previous similar messages [ 6716.554693] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6716.572454] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6716.581507] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6716.592604] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6716.635325] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6720.593329] Lustre: server umount lustre-OST0001 complete [ 6722.025148] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6722.033215] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6722.038890] LustreError: Skipped 1 previous similar message [ 6722.058252] LustreError: 221433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6722.058544] Lustre: Skipped 2 previous similar messages [ 6725.600408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6725.605542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.619242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6727.649651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6727.654429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6727.655336] LustreError: 220409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6727.655344] LustreError: 220409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6727.695153] Lustre: Skipped 1 previous similar message [ 6731.018203] Lustre: server umount lustre-OST0000 complete [ 6736.867626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6736.883123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6736.914566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6740.692599] Lustre: server umount lustre-MDT0000 complete [ 6743.008797] LustreError: 218586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6743.036643] LustreError: 218586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6744.400463] LustreError: 218567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648253 with bad export cookie 17090865581196185099 [ 6744.417355] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6744.425896] LustreError: 218567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6744.815971] Lustre: server umount lustre-MDT0001 complete [ 6754.612430] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 6757.301091] Key type lgssc unregistered [ 6757.657704] LNet: 223680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6757.664687] LNetError: 223680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6757.684658] LNet: Removed LNI 192.168.206.112@tcp [ 6758.385249] Key type .llcrypt unregistered [ 6758.387687] Key type ._llcrypt unregistered [ 6795.747481] Key type ._llcrypt registered [ 6795.749166] Key type .llcrypt registered [ 6795.855826] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 6796.565747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6796.603996] alg: No test for adler32 (adler32-zlib) [ 6797.585698] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 6797.922259] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 6799.663271] Key type lgssc registered [ 6800.885498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6810.296527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6810.317774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6811.607083] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6811.662616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6811.836171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6817.558067] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6827.351346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6827.550764] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6827.635031] Lustre: 225862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6828.099544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6832.679653] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6839.743882] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6846.438668] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6852.836443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6852.953860] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6853.119218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6858.519874] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6863.027620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6866.156572] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6870.223098] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6873.671466] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6880.214353] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6887.881728] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6903.770383] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6903.785658] Lustre: Skipped 1 previous similar message [ 6925.713343] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6925.726271] Lustre: Skipped 2 previous similar messages [ 6961.472327] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6961.480464] Lustre: Skipped 4 previous similar messages [ 7033.551178] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7033.559534] Lustre: Skipped 7 previous similar messages [ 7042.017860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7042.031763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7042.045388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7042.528417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7042.541525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7042.555286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7046.762818] Lustre: server umount lustre-OST0000 complete [ 7052.770996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.780838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7052.793228] Lustre: Skipped 1 previous similar message [ 7057.889191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7057.903210] Lustre: Skipped 1 previous similar message [ 7063.008853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7063.021285] Lustre: Skipped 1 previous similar message [ 7065.055161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7065.373800] Lustre: server umount lustre-MDT0000 complete [ 7069.912425] LustreError: 228652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648578 with bad export cookie 2878733954971686215 [ 7069.920698] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7069.930631] LustreError: 228652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7070.428829] Lustre: server umount lustre-MDT0001 complete [ 7083.735476] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 7087.903836] Key type lgssc unregistered [ 7088.317178] LNet: 230682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7088.328916] LNetError: 230682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7088.370289] LNet: Removed LNI 192.168.206.112@tcp [ 7089.744435] Key type .llcrypt unregistered [ 7089.747396] Key type ._llcrypt unregistered [ 7116.295850] Key type ._llcrypt registered [ 7116.300128] Key type .llcrypt registered [ 7116.424798] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 7117.842890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7117.926670] alg: No test for adler32 (adler32-zlib) [ 7118.939719] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 7119.248418] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 7120.967317] Key type lgssc registered [ 7122.187528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7130.340564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7130.376565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7132.170162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7136.761652] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7142.374484] LustreError: 232092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7142.389745] LustreError: 232092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7143.843580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7144.273136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7147.867478] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7154.952891] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7160.148810] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7166.685024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7167.047042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7172.640554] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7177.272871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7179.192914] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7200.738384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7200.754618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7200.768412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7202.792767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7202.795705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7202.838554] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7205.102142] Lustre: server umount lustre-OST0000 complete [ 7210.977555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7210.985908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7211.024960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7211.031878] Lustre: Skipped 1 previous similar message [ 7213.025414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7213.028402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7215.314630] Lustre: server umount lustre-MDT0000 complete [ 7218.149934] LustreError: 232087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7218.169090] LustreError: 232087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7218.681702] LustreError: 232072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648727 with bad export cookie 2019970760509004309 [ 7218.684070] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7218.690081] LustreError: 232072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7219.016522] Lustre: server umount lustre-MDT0001 complete [ 7228.738849] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 7231.681392] Key type lgssc unregistered [ 7231.959650] LNet: 235777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7231.965788] LNetError: 235777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7232.997243] LNet: Removed LNI 192.168.206.112@tcp [ 7233.849059] Key type .llcrypt unregistered [ 7233.851958] Key type ._llcrypt unregistered [ 7256.513075] Key type ._llcrypt registered [ 7256.514558] Key type .llcrypt registered [ 7256.677069] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 7257.848600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7257.889878] alg: No test for adler32 (adler32-zlib) [ 7258.975120] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 7259.191263] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 7260.839302] Key type lgssc registered [ 7262.041531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7533.807456] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 7537.611070] Key type lgssc unregistered [ 7538.094901] LNet: 237806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7538.107734] LNetError: 237806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7538.142310] LNet: Removed LNI 192.168.206.112@tcp [ 7539.531396] Key type .llcrypt unregistered [ 7539.535385] Key type ._llcrypt unregistered [ 7570.992580] Key type ._llcrypt registered [ 7570.994401] Key type .llcrypt registered [ 7571.100802] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 7586.761556] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 7587.781470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7587.898066] alg: No test for adler32 (adler32-zlib) [ 7588.984627] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 7589.268092] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 7590.911314] Key type lgssc registered [ 7591.992381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7627.861571] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7333 sec ========== 18:58:53 (1788649133) [ 7629.545994] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:58:55 (1788649135) === [ 7632.978941] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:58:58 (1788649138) === [ 7672.434677] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 7674.776952] Key type lgssc unregistered [ 7675.040588] LNet: 243039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7675.046534] LNetError: 243039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7675.063604] LNet: Removed LNI 192.168.206.112@tcp [ 7675.827297] Key type .llcrypt unregistered [ 7675.829916] Key type ._llcrypt unregistered