[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487751159 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002369] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007452] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010122] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.013076] *** VALIDATE selinux *** [ 0.021792] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026472] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027182] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029132] *** VALIDATE tmpfs *** [ 0.031428] *** VALIDATE proc *** [ 0.032265] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034286] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037032] Spectre V2 : User space: Vulnerable [ 0.038012] Speculative Store Bypass: Vulnerable [ 0.041527] debug: unmapping init [mem 0xffffffffac859000-0xffffffffac860fff] [ 0.043184] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044710] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047015] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050011] ... fixed-purpose events: 3 [ 0.051015] ... event mask: 000000070000000f [ 0.052315] rcu: Hierarchical SRCU implementation. [ 0.054574] smp: Bringing up secondary CPUs ... [ 0.055613] x86: Booting SMP configuration: [ 0.056027] .... node #0, CPUs: #1 #2 #3 [ 0.060048] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.152017] node 0 deferred pages initialised in 86ms [ 0.155008] devtmpfs: initialized [ 0.156251] x86/mm: Memory block size: 128MB [ 0.158711] gcov: version magic: 0x41383552 [ 0.160221] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162305] pinctrl core: initialized pinctrl subsystem [ 0.163262] [ 0.163801] ************************************************************* [ 0.164015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165010] ** ** [ 0.166012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167009] ** ** [ 0.168009] ** This means that this kernel is built to expose internal ** [ 0.169010] ** IOMMU data structures, which may compromise security on ** [ 0.170024] ** your system. ** [ 0.171008] ** ** [ 0.172011] ** If you see this message and you are not debugging the ** [ 0.173015] ** kernel, report this immediately to your vendor! ** [ 0.174016] ** ** [ 0.175015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176018] ************************************************************* [ 0.177650] NET: Registered protocol family 16 [ 0.178456] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.180065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.181440] cpuidle: using governor menu [ 0.184641] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.187525] PCI: Using configuration type 1 for base access [ 0.190134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.201103] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.202020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.205049] cryptd: max_cpu_qlen set to 1000 [ 0.208116] ACPI: Added _OSI(Module Device) [ 0.209013] ACPI: Added _OSI(Processor Device) [ 0.210014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.211016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.215091] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218235] ACPI: Interpreter enabled [ 0.219048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.220009] ACPI: Using IOAPIC for interrupt routing [ 0.221109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222352] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.231525] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.232046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236211] acpiphp: Slot [2] registered [ 0.237096] acpiphp: Slot [5] registered [ 0.237997] acpiphp: Slot [6] registered [ 0.238066] acpiphp: Slot [7] registered [ 0.238855] acpiphp: Slot [8] registered [ 0.239058] acpiphp: Slot [9] registered [ 0.239938] acpiphp: Slot [10] registered [ 0.240110] acpiphp: Slot [3] registered [ 0.241093] acpiphp: Slot [4] registered [ 0.242081] acpiphp: Slot [11] registered [ 0.243087] acpiphp: Slot [12] registered [ 0.244084] acpiphp: Slot [13] registered [ 0.245111] acpiphp: Slot [14] registered [ 0.246097] acpiphp: Slot [15] registered [ 0.247108] acpiphp: Slot [16] registered [ 0.248091] acpiphp: Slot [17] registered [ 0.249086] acpiphp: Slot [18] registered [ 0.250093] acpiphp: Slot [19] registered [ 0.251106] acpiphp: Slot [20] registered [ 0.252139] acpiphp: Slot [21] registered [ 0.253104] acpiphp: Slot [22] registered [ 0.254092] acpiphp: Slot [23] registered [ 0.255090] acpiphp: Slot [24] registered [ 0.256108] acpiphp: Slot [25] registered [ 0.257098] acpiphp: Slot [26] registered [ 0.258096] acpiphp: Slot [27] registered [ 0.259135] acpiphp: Slot [28] registered [ 0.260137] acpiphp: Slot [29] registered [ 0.261080] acpiphp: Slot [30] registered [ 0.262125] acpiphp: Slot [31] registered [ 0.264070] PCI host bridge to bus 0000:00 [ 0.265017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.268025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.271025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.272024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.274110] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.275017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.276220] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.277946] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.279034] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.284945] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.286547] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.288019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.289020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.290018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.292780] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.294854] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.296050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.297895] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.300916] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.309016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.314000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.319386] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.326018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.331017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.345027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.358422] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.368018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.373015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.387016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.400043] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.413034] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.435026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.498021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.515936] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.521017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.527019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.540021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.548363] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.553015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.558000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.570022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.580110] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.585016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.590020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.602015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.613233] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.616551] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.619791] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.623621] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.626271] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.632122] iommu: Default domain type: Passthrough [ 0.633566] SCSI subsystem initialized [ 0.635159] ACPI: bus type USB registered [ 0.636136] usbcore: registered new interface driver usbfs [ 0.638120] usbcore: registered new interface driver hub [ 0.640145] usbcore: registered new device driver usb [ 0.641197] pps_core: LinuxPPS API ver. 1 registered [ 0.643018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.646120] PTP clock support registered [ 0.648119] EDAC MC: Ver: 3.0.0 [ 0.650177] PCI: Using ACPI for IRQ routing [ 0.653363] NetLabel: Initializing [ 0.655020] NetLabel: domain hash size = 128 [ 0.657015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.661107] NetLabel: unlabeled traffic allowed by default [ 0.664164] vgaarb: loaded [ 0.666258] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.669024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.676541] clocksource: Switched to clocksource kvm-clock [ 0.803621] VFS: Disk quotas dquot_6.6.0 [ 0.806093] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.809185] *** VALIDATE ramfs *** [ 0.810907] *** VALIDATE hugetlbfs *** [ 0.813271] pnp: PnP ACPI init [ 0.817789] pnp: PnP ACPI: found 6 devices [ 0.835336] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.838931] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.840587] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.842240] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.844096] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.845970] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.848129] NET: Registered protocol family 2 [ 0.850240] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.855772] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.859479] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.864982] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.868600] TCP: Hash tables configured (established 65536 bind 65536) [ 0.871727] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.874915] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.877984] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.881443] NET: Registered protocol family 1 [ 0.884271] RPC: Registered named UNIX socket transport module. [ 0.886365] RPC: Registered udp transport module. [ 0.887915] RPC: Registered tcp transport module. [ 0.890666] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.892836] NET: Registered protocol family 44 [ 0.894490] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.896861] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.899132] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.901631] PCI: CLS 0 bytes, default 64 [ 0.903422] Unpacking initramfs... [ 2.250902] debug: unmapping init [mem 0xffff9f94fcc54000-0xffff9f94fffbffff] [ 2.254904] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.257513] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.260496] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.745491] Initialise system trusted keyrings [ 2.747212] Key type blacklist registered [ 2.749418] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.756794] zbud: loaded [ 2.759862] *** VALIDATE nfs *** [ 2.760802] *** VALIDATE nfs4 *** [ 2.761797] pstore: using deflate compression [ 2.764691] Platform Keyring initialized [ 2.858833] NET: Registered protocol family 38 [ 2.859822] Key type asymmetric registered [ 2.861032] Asymmetric key parser 'x509' registered [ 2.862521] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.864826] io scheduler mq-deadline registered [ 2.866022] io scheduler kyber registered [ 2.867329] io scheduler bfq registered [ 2.868446] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.870405] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.872277] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.874187] ACPI: Power Button [PWRF] [ 2.877910] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.882523] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.892492] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.898651] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.915136] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.944276] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.973556] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.978471] Non-volatile memory driver v1.3 [ 2.979586] Linux agpgart interface v0.103 [ 3.001939] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.003669] vda: detected capacity change from 0 to 68624384 [ 3.025590] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.028881] vdb: detected capacity change from 0 to 1073741824 [ 3.049921] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.054415] vdc: detected capacity change from 0 to 2621440000 [ 3.074134] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.077177] vdd: detected capacity change from 0 to 2621440000 [ 3.102665] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.107120] vde: detected capacity change from 0 to 4294967296 [ 3.127528] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.130472] vdf: detected capacity change from 0 to 4294967296 [ 3.138199] libphy: Fixed MDIO Bus: probed [ 3.148393] usbcore: registered new interface driver usbserial_generic [ 3.152035] usbserial: USB Serial support registered for generic [ 3.154596] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.159092] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.161223] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.164050] mousedev: PS/2 mouse device common for all mice [ 3.166903] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.169662] rtc_cmos 00:05: RTC can wake from S4 [ 3.174298] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.174441] rtc_cmos 00:05: registered as rtc0 [ 3.179746] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.182424] intel_pstate: CPU model not supported [ 3.184454] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.189351] hid: raw HID events driver (C) Jiri Kosina [ 3.191322] usbcore: registered new interface driver usbhid [ 3.193118] usbhid: USB HID core driver [ 3.194486] drop_monitor: Initializing network drop monitor service [ 3.196655] Initializing XFRM netlink socket [ 3.198341] NET: Registered protocol family 10 [ 3.200878] Segment Routing with IPv6 [ 3.202156] NET: Registered protocol family 17 [ 3.204119] mpls_gso: MPLS GSO support [ 3.208340] RAS: Correctable Errors collector initialized. [ 3.210184] AVX version of gcm_enc/dec engaged. [ 3.211531] AES CTR mode by8 optimization enabled [ 3.269194] sched_clock: Marking stable (3269177301, 0)->(4299352784, -1030175483) [ 3.272797] registered taskstats version 1 [ 3.274556] Loading compiled-in X.509 certificates [ 3.276735] zswap: loaded using pool lzo/zbud [ 3.294253] Key type big_key registered [ 3.306291] Key type encrypted registered [ 3.307748] ima: No TPM chip found, activating TPM-bypass! [ 3.309474] ima: Allocated hash algorithm: sha1 [ 3.310996] ima: No architecture policies found [ 3.312387] evm: Initialising EVM extended attributes: [ 3.313829] evm: security.selinux [ 3.314655] evm: security.ima [ 3.315468] evm: security.capability [ 3.316499] evm: HMAC attrs: 0x1 [ 3.318257] rtc_cmos 00:05: setting system clock to 2026-01-22 10:47:57 UTC (1769078877) [ 3.322849] debug: unmapping init [mem 0xffffffffad803000-0xffffffffad9fffff] [ 3.325244] debug: unmapping init [mem 0xffffffffac582000-0xffffffffac858fff] [ 3.333126] Write protecting the kernel read-only data: 28672k [ 3.335545] debug: unmapping init [mem 0xffffffffaac03000-0xffffffffaadfffff] [ 3.337795] debug: unmapping init [mem 0xffffffffab514000-0xffffffffab5fffff] [ 3.361627] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.368755] systemd[1]: Detected virtualization kvm. [ 3.371537] systemd[1]: Detected architecture x86-64. [ 3.374441] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.404285] systemd[1]: No hostname configured. [ 3.406861] systemd[1]: Set hostname to . [ 3.409792] random: systemd: uninitialized urandom read (16 bytes read) [ 3.412407] systemd[1]: Initializing machine ID from random generator. [ 3.551251] random: systemd: uninitialized urandom read (16 bytes read) [ 3.554336] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.561688] random: systemd: uninitialized urandom read (16 bytes read) [ 3.565833] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.575435] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.227794] device-mapper: uevent: version 1.0.3 [ 4.229782] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ 4.971471] random: fast init done [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.040679] virtio_net virtio0 ens2: renamed from eth0 [ 5.166542] scsi host0: ata_piix [ 5.222884] scsi host1: ata_piix [ 5.224317] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.226824] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.505409] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.820907] random: crng init done [ 9.822057] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.259515] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.433193] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.705357] SELinux: Disabled at runtime. [ 11.768252] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.778739] systemd[1]: Detected virtualization kvm. [ 11.781182] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.305728] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.308982] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.313505] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.318360] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.321813] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.328600] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.335038] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.412570] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.847578] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.104335] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.104854] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.206858] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.228287] EDAC sbridge: Ver: 1.1.2 [ 14.823509] Key type dns_resolver registered [ 15.138058] NFS: Registering the id_resolver key type [ 15.140220] Key type id_resolver registered [ 15.142018] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg317-server login: [ 40.024775] libcfs: loading out-of-tree module taints kernel. [ 40.044512] Key type ._llcrypt registered [ 40.046070] Key type .llcrypt registered [ 40.098471] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 56.717979] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 57.874102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 57.891075] alg: No test for adler32 (adler32-zlib) [ 59.561522] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 60.564995] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 62.400622] Key type lgssc registered [ 63.923095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 84.245435] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 85.729204] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.890947] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 100.083928] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.366503] hrtimer: interrupt took 3528061 ns [ 108.051656] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.954149] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 139.500728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 139.575551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 139.598401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 140.845979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 140.894115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 141.036594] Lustre: lustre-MDT0000: new disk, initializing [ 141.134117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 141.149644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 144.031459] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 156.342193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 156.414972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 156.467835] Lustre: 6510:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 156.507119] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 156.511670] Lustre: Skipped 1 previous similar message [ 156.530708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 156.596290] Lustre: lustre-MDT0001: new disk, initializing [ 156.659464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 156.685508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 156.700697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 161.259587] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.361315] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 174.630458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 174.720535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 174.976915] Lustre: lustre-OST0000: new disk, initializing [ 174.981408] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 175.080982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 175.661869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 175.669391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 175.786135] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 181.284038] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.307952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 201.371973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 201.461550] Lustre: lustre-OST0001: new disk, initializing [ 201.464553] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 201.513295] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.796427] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.529492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 210.552217] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 210.668899] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 223.561480] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.901729] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.040763] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing check_logdir /tmp/testlogs/ [ 258.264804] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing yml_node [ 262.939298] Lustre: DEBUG MARKER: Client: 2.17.50.3 [ 265.718889] Lustre: DEBUG MARKER: MDS: 2.17.50.3 [ 268.236682] Lustre: DEBUG MARKER: OSS: 2.17.50.3 [ 270.890103] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jan 22 05:52:22 EST 2026 [ 289.405364] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 290.962983] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 308.192989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.194140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.206596] Lustre: Skipped 3 previous similar messages [ 308.224558] Lustre: Skipped 3 previous similar messages [ 311.668493] LustreError: 12219:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 311.880446] Lustre: server umount lustre-MDT0000 complete [ 315.991205] LustreError: 6503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079190 with bad export cookie 16363013216127987531 [ 315.999360] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 318.448302] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.448985] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.450465] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 318.465618] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 318.485366] Lustre: Skipped 1 previous similar message [ 320.481767] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.483588] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.501923] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 320.518625] Lustre: Skipped 2 previous similar messages [ 322.244297] LustreError: 12421:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 322.249897] LustreError: 12421:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 322.464377] Lustre: server umount lustre-MDT0001 complete [ 326.170226] LustreError: 12623:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 326.177625] LustreError: 12623:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 326.323480] Lustre: server umount lustre-OST0000 complete [ 329.946104] LustreError: 12824:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 329.953429] LustreError: 12824:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 330.163961] Lustre: server umount lustre-OST0001 complete [ 336.758398] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 345.736044] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 355.016900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 360.965681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 366.812583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 373.084501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 379.864781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 379.962819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.216599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 380.274865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 380.375557] Lustre: lustre-MDT0000: new disk, initializing [ 380.480200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 380.495636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 384.428554] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 393.423992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 393.551606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 393.686226] Lustre: 15009:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 393.726635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 393.728883] Lustre: Skipped 1 previous similar message [ 393.841627] Lustre: lustre-MDT0001: new disk, initializing [ 393.913647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 393.944838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 393.965682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 397.813622] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 405.340236] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 408.804805] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 415.490986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 415.555548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 415.803078] Lustre: lustre-OST0000: new disk, initializing [ 415.807032] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 415.900691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.916148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 416.929203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 417.067531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 422.451827] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 429.199887] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 432.911815] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 433.092165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 436.643952] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 436.783629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 442.850836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 442.851325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 442.876297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 442.892875] Lustre: Skipped 1 previous similar message [ 444.534193] LustreError: 17951:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 444.536878] LustreError: 17951:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 444.669936] Lustre: server umount lustre-OST0000 complete [ 450.529749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 450.541320] LustreError: Skipped 1 previous similar message [ 450.548734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.562662] Lustre: Skipped 1 previous similar message [ 453.091538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 453.094053] Lustre: Skipped 2 previous similar messages [ 454.240121] LustreError: 18152:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 454.247554] LustreError: 18152:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 454.363490] Lustre: server umount lustre-MDT0000 complete [ 457.694174] LustreError: 16768:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079331 with bad export cookie 16363013216127989064 [ 457.701123] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 457.709880] LustreError: 16768:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 457.979637] Lustre: server umount lustre-MDT0001 complete [ 460.902119] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 05:55:34 (1769079334) [ 466.876684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.280048] LustreError: 18879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.367748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 471.040283] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 472.544978] LustreError: 18879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.677977] LustreError: 18879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 478.786867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 482.840712] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 488.384515] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 491.517770] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 498.328442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 504.417854] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 509.378624] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 522.215433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 522.227500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.237390] Lustre: Skipped 1 previous similar message [ 522.240776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 522.253358] Lustre: Skipped 1 previous similar message [ 527.942524] LustreError: 21145:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 527.950772] LustreError: 21145:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 528.110874] Lustre: server umount lustre-OST0000 complete [ 532.451768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.465993] Lustre: Skipped 1 previous similar message [ 537.738731] Lustre: server umount lustre-MDT0000 complete [ 540.620134] LustreError: 20496:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079414 with bad export cookie 16363013216127989946 [ 540.623701] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 540.627427] LustreError: 20496:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 540.837891] Lustre: server umount lustre-MDT0001 complete [ 550.100654] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 552.303337] Key type lgssc unregistered [ 552.590564] LNet: 22081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 552.595137] LNetError: 22081:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 552.616940] LNet: Removed LNI 192.168.203.117@tcp [ 553.445392] Key type .llcrypt unregistered [ 553.447668] Key type ._llcrypt unregistered [ 571.224174] Key type ._llcrypt registered [ 571.227994] Key type .llcrypt registered [ 571.350376] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 572.178857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 572.231772] alg: No test for adler32 (adler32-zlib) [ 573.212412] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 573.394930] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 575.048193] Key type lgssc registered [ 575.958856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 581.937834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 581.958514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.262238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 586.234702] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 592.869289] LustreError: 23454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.886920] LustreError: 23454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 593.025633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 593.482410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 596.912750] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 601.173867] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 604.082610] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 609.324412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 609.569738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.425318] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 618.710613] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 628.015887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 637.422775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.426360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 637.427600] Lustre: Skipped 1 previous similar message [ 637.432713] Lustre: Skipped 1 previous similar message [ 641.222153] LustreError: 26139:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 641.388249] Lustre: server umount lustre-OST0000 complete [ 644.579275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 644.587241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.599297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.648973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.650720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.606901] LustreError: 26342:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 650.614468] LustreError: 26342:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 650.760594] Lustre: server umount lustre-MDT0000 complete [ 652.770166] LustreError: 26053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.780353] LustreError: 26053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 653.508145] LustreError: 23436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079527 with bad export cookie 12313590170097453706 [ 653.515583] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 653.519284] LustreError: 23436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 653.655291] LustreError: 26543:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 653.658026] LustreError: 26543:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 653.841674] Lustre: server umount lustre-MDT0001 complete [ 662.098396] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 664.136636] Key type lgssc unregistered [ 664.353648] LNet: 27075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.357575] LNetError: 27075:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.368525] LNet: Removed LNI 192.168.203.117@tcp [ 664.995840] Key type .llcrypt unregistered [ 665.002530] Key type ._llcrypt unregistered [ 682.472970] Key type ._llcrypt registered [ 682.475571] Key type .llcrypt registered [ 682.536963] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 683.191613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 683.306701] alg: No test for adler32 (adler32-zlib) [ 684.299813] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 684.487770] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 686.120201] Key type lgssc registered [ 686.877595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 692.435208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 692.459779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.778497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 697.259814] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 702.958806] LustreError: 28463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.970663] LustreError: 28463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 703.908772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 704.232144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 707.126471] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 711.998568] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 715.030371] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 724.972533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 725.193064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.915809] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 733.927913] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 739.500432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 750.051341] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 750.056171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.059350] LustreError: Skipped 1 previous similar message [ 750.059924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 750.084489] Lustre: Skipped 1 previous similar message [ 753.980162] LustreError: 31076:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 754.058309] Lustre: server umount lustre-OST0000 complete [ 760.294054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.299105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 760.302402] Lustre: Skipped 1 previous similar message [ 760.311141] Lustre: Skipped 1 previous similar message [ 762.780727] LustreError: 31278:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 762.787292] LustreError: 31278:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 762.843651] Lustre: server umount lustre-MDT0000 complete [ 765.367521] LustreError: 28443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079639 with bad export cookie 6884948523354631759 [ 765.371296] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 765.373965] LustreError: 28443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 765.500734] LustreError: 31480:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 765.512813] LustreError: 31480:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 765.637991] Lustre: server umount lustre-MDT0001 complete [ 772.738216] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 774.694984] Key type lgssc unregistered [ 774.929607] LNet: 32012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.933637] LNetError: 32012:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.942669] LNet: Removed LNI 192.168.203.117@tcp [ 775.447181] Key type .llcrypt unregistered [ 775.450963] Key type ._llcrypt unregistered [ 791.559045] Key type ._llcrypt registered [ 791.560632] Key type .llcrypt registered [ 791.637726] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 792.281932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 792.331417] alg: No test for adler32 (adler32-zlib) [ 793.319846] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 793.487375] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 795.112122] Key type lgssc registered [ 795.752194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 800.420879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 800.442664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 801.845602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.012279] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 809.120345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 809.294958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 811.241377] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 814.576950] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 817.261340] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 821.383857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 821.585040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 824.866543] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 826.733685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 828.734236] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 839.138137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 839.141888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 839.155137] Lustre: Skipped 1 previous similar message [ 839.164018] Lustre: Skipped 1 previous similar message [ 840.707271] LustreError: 35642:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 840.821389] Lustre: server umount lustre-OST0000 complete [ 844.260060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.278174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 844.281015] Lustre: Skipped 1 previous similar message [ 849.295987] LustreError: 35842:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 849.301678] LustreError: 35842:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 849.370886] Lustre: server umount lustre-MDT0000 complete [ 849.381436] LustreError: 33397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.394214] LustreError: 33397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 851.371055] LustreError: 33376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079725 with bad export cookie 8776637503270551077 [ 851.376301] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 851.376985] LustreError: 33376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 851.467068] LustreError: 36045:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 851.469682] LustreError: 36045:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 851.557513] Lustre: server umount lustre-MDT0001 complete [ 857.172254] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 858.584340] Key type lgssc unregistered [ 858.736676] LNet: 36577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.742937] LNetError: 36577:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.814627] LNet: Removed LNI 192.168.203.117@tcp [ 860.338452] Key type .llcrypt unregistered [ 860.342349] Key type ._llcrypt unregistered [ 875.533465] Key type ._llcrypt registered [ 875.535577] Key type .llcrypt registered [ 875.618686] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 876.286984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 876.305364] alg: No test for adler32 (adler32-zlib) [ 877.303351] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 877.460614] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 879.096142] Key type lgssc registered [ 879.834870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 884.496446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 884.507917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 885.758785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 888.151969] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 894.136288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 894.402611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 897.422182] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 900.874769] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 903.078557] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 907.185211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 907.401128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 910.519299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 910.587161] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 913.789717] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 925.157587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.158196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 925.167516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 926.294931] Lustre: lustre-OST0000: Not available for connect from 192.168.203.17@tcp (stopping) [ 926.298286] Lustre: Skipped 1 previous similar message [ 928.458181] LustreError: 40195:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 928.503850] Lustre: server umount lustre-OST0000 complete [ 945.057908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 945.059765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 945.065523] Lustre: Skipped 2 previous similar messages [ 945.073774] Lustre: Skipped 1 previous similar message [ 950.243978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 950.252351] Lustre: Skipped 1 previous similar message [ 950.255207] LustreError: 40571:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 950.261523] LustreError: 40571:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 950.310779] Lustre: server umount lustre-MDT0000 complete [ 952.536020] LustreError: 37930:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079826 with bad export cookie 11095486493019925480 [ 952.537476] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 952.546649] LustreError: 37930:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 958.679536] LustreError: 40773:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 958.683252] LustreError: 40773:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 958.767570] Lustre: server umount lustre-MDT0001 complete [ 965.155598] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 966.945170] Key type lgssc unregistered [ 967.156540] LNet: 41306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.164961] LNetError: 41306:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.229761] LNet: Removed LNI 192.168.203.117@tcp [ 968.806304] Key type .llcrypt unregistered [ 968.807573] Key type ._llcrypt unregistered [ 985.345382] Key type ._llcrypt registered [ 985.346463] Key type .llcrypt registered [ 985.422245] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 986.037711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 986.047368] alg: No test for adler32 (adler32-zlib) [ 986.972199] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 987.107471] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 988.744165] Key type lgssc registered [ 989.446940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 995.015862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 995.030551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 996.408125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 999.340372] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1005.115601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1005.317991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1008.215334] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1012.000425] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1014.780736] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1019.105820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1019.333595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.422170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1023.197553] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1026.786262] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1036.258153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1036.260866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.268670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1039.457154] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.17@tcp (stopping) [ 1040.870076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1040.871162] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.075597] LustreError: 44923:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1042.127832] Lustre: server umount lustre-MDT0000 complete [ 1044.456042] LustreError: 42658:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079918 with bad export cookie 11063327583180903601 [ 1044.462474] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1044.465381] LustreError: 42658:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1044.557902] LustreError: 42673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1044.572308] LustreError: 42673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1044.588725] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.17@tcp (stopping) [ 1044.590943] Lustre: Skipped 2 previous similar messages [ 1045.985615] LustreError: 42673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.991385] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.006721] Lustre: Skipped 1 previous similar message [ 1049.679128] LustreError: 44906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1049.679245] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.17@tcp (stopping) [ 1049.703020] Lustre: Skipped 1 previous similar message [ 1050.657960] LustreError: 45127:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1050.662239] LustreError: 45127:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1050.809547] Lustre: server umount lustre-MDT0001 complete [ 1078.240216] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1078.277596] LustreError: 45351:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1078.288435] LustreError: 45351:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1078.407219] Lustre: server umount lustre-OST0000 complete [ 1088.488870] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 1090.099467] Key type lgssc unregistered [ 1090.257536] LNet: 46190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.261490] LNetError: 46190:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1090.272843] LNet: Removed LNI 192.168.203.117@tcp [ 1090.717481] Key type .llcrypt unregistered [ 1090.719653] Key type ._llcrypt unregistered [ 1105.772567] Key type ._llcrypt registered [ 1105.775220] Key type .llcrypt registered [ 1105.845763] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1106.560258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1106.572491] alg: No test for adler32 (adler32-zlib) [ 1107.582952] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 1107.708496] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 1109.328156] Key type lgssc registered [ 1110.084935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1114.740347] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1114.755751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1130.208370] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1135.328586] LustreError: 47535:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.117@tcp: failed processing log, type 4: rc = -110 [ 1161.952228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1165.122950] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1168.088211] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1335.033869] LustreError: 48253:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1335.099247] Lustre: server umount lustre-OST0000 complete [ 1343.444827] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 1344.850518] Key type lgssc unregistered [ 1345.023285] LNet: 49088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.032902] LNetError: 49088:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.054724] LNet: Removed LNI 192.168.203.117@tcp [ 1345.547193] Key type .llcrypt unregistered [ 1345.549233] Key type ._llcrypt unregistered [ 1358.348687] Key type ._llcrypt registered [ 1358.349802] Key type .llcrypt registered [ 1358.401304] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1358.922932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1358.968890] alg: No test for adler32 (adler32-zlib) [ 1359.883873] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 1360.003534] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 1361.608155] Key type lgssc registered [ 1362.290318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1365.631200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1365.639312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.862797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1368.715321] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1372.603213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1372.785070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1374.670890] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1377.412428] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1379.482862] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1383.001524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1383.178240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1386.111868] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1389.369670] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1390.695098] LustreError: 52702:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1390.743029] Lustre: server umount lustre-OST0000 complete [ 1392.561517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.564843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1392.567481] Lustre: Skipped 1 previous similar message [ 1397.731934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1397.735467] Lustre: Skipped 2 previous similar messages [ 1398.728048] LustreError: 52902:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1398.733446] LustreError: 52902:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1398.807431] Lustre: server umount lustre-MDT0000 complete [ 1400.475232] LustreError: 50441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080274 with bad export cookie 7130876089227826518 [ 1400.480048] LustreError: 50441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1400.481138] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1400.550470] LustreError: 53104:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1400.553083] LustreError: 53104:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1400.619528] Lustre: server umount lustre-MDT0001 complete [ 1405.249448] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 1406.533092] Key type lgssc unregistered [ 1406.685474] LNet: 53636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.688762] LNetError: 53636:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.695548] LNet: Removed LNI 192.168.203.117@tcp [ 1407.057220] Key type .llcrypt unregistered [ 1407.059255] Key type ._llcrypt unregistered [ 1418.520943] Key type ._llcrypt registered [ 1418.525168] Key type .llcrypt registered [ 1418.583980] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1418.939768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1418.978267] alg: No test for adler32 (adler32-zlib) [ 1419.873118] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 1419.976585] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 1421.578860] Key type lgssc registered [ 1422.140464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1425.874472] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1425.888088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1441.376294] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1446.496191] LustreError: 54981:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.117@tcp: failed processing log, type 4: rc = -110 [ 1473.120164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.119124] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1478.645885] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1482.041178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.315266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.361952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1484.380761] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1487.329659] LustreError: 55007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.341497] LustreError: 55007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1488.359812] LustreError: 55007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.382180] LustreError: 55007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1489.075383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1489.289796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1491.162420] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1493.470729] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1495.060189] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1497.570519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1497.573873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.580497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1499.620678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1499.621551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1499.622920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1504.741579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1504.745728] Lustre: Skipped 1 previous similar message [ 1509.864673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1509.869218] Lustre: Skipped 1 previous similar message [ 1510.368247] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1510.395107] LustreError: 57255:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1510.447671] Lustre: server umount lustre-OST0000 complete [ 1530.337769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1530.337868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1530.338602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1530.338607] Lustre: Skipped 1 previous similar message [ 1530.343908] Lustre: Skipped 1 previous similar message [ 1532.551121] LustreError: 57635:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1532.558330] LustreError: 57635:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1532.597165] Lustre: server umount lustre-MDT0000 complete [ 1534.299548] LustreError: 54990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080408 with bad export cookie 8706512394878229544 [ 1534.302681] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1534.308608] LustreError: 54990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1540.417721] LustreError: 57840:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1540.422887] LustreError: 57840:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1540.504386] Lustre: server umount lustre-MDT0001 complete [ 1545.406948] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 1546.715152] Key type lgssc unregistered [ 1546.873607] LNet: 58372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.878652] LNetError: 58372:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.890618] LNet: Removed LNI 192.168.203.117@tcp [ 1547.315857] Key type .llcrypt unregistered [ 1547.317370] Key type ._llcrypt unregistered [ 1559.086481] Key type ._llcrypt registered [ 1559.087735] Key type .llcrypt registered [ 1559.140641] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1559.514823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1559.524659] alg: No test for adler32 (adler32-zlib) [ 1560.418218] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 1560.515813] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 1562.104187] Key type lgssc registered [ 1562.634073] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1566.564381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1566.577587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1567.837684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.688072] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1574.166831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1574.378830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1576.167232] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1578.776828] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1580.505433] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1583.951518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1584.138378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.527359] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1589.003567] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1592.361635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1602.531013] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1602.532551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1602.536182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1602.541082] Lustre: Skipped 1 previous similar message [ 1607.652037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1607.654906] Lustre: Skipped 1 previous similar message [ 1608.113109] LustreError: 62023:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1608.157193] Lustre: server umount lustre-OST0000 complete [ 1610.209763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1610.212170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1610.220668] Lustre: Skipped 1 previous similar message [ 1610.224342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1612.769985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1612.771664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1616.022496] LustreError: 62224:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1616.026570] LustreError: 62224:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1616.061333] Lustre: server umount lustre-MDT0000 complete [ 1617.787673] LustreError: 59737:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080491 with bad export cookie 1621154727882071612 [ 1617.788348] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1617.792633] LustreError: 59737:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1617.908301] LustreError: 62427:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1617.911654] LustreError: 62427:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1618.007747] Lustre: server umount lustre-MDT0001 complete [ 1622.766906] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 1623.976976] Key type lgssc unregistered [ 1624.145530] LNet: 62958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.148554] LNetError: 62958:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1624.167663] LNet: Removed LNI 192.168.203.117@tcp [ 1624.539271] Key type .llcrypt unregistered [ 1624.540332] Key type ._llcrypt unregistered [ 1636.103546] Key type ._llcrypt registered [ 1636.105020] Key type .llcrypt registered [ 1636.152279] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1636.537680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1636.563954] alg: No test for adler32 (adler32-zlib) [ 1637.457171] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 1637.565128] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 1639.152156] Key type lgssc registered [ 1639.595823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1643.314600] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 06:15:16 (1769080516) [ 1646.616290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1646.628940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1647.861907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.525415] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1653.515340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1653.725552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1655.419868] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1657.735924] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1659.312375] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1662.286813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1662.495531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.055971] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1667.558874] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1673.770591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1680.537742] Lustre: Failing over lustre-MDT0000 [ 1680.657992] LustreError: 66886:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1680.730593] Lustre: server umount lustre-MDT0000 complete [ 1683.035918] LustreError: 64615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1683.049360] LustreError: 64615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1683.940413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1683.944625] LustreError: 64621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1683.951870] Lustre: Skipped 2 previous similar messages [ 1683.961889] LustreError: 64621:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1685.022408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.099736] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1685.164109] Lustre: *** cfs_fail_loc=135, val=0*** [ 1685.214770] LustreError: 67396:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1685.217218] LustreError: 67396:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.117@tcp: cfg command failed: rc = -2 [ 1685.219663] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1685.219663] [ 1685.222506] LustreError: MGC192.168.203.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1685.226358] LustreError: 67384:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1685.230839] LustreError: 67384:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1685.232960] LustreError: 67384:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 1685.239729] LustreError: 67384:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 1685.254960] Lustre: server umount lustre-MDT0000 complete [ 1685.256618] LustreError: 67384:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1688.138430] LustreError: 66357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1689.241301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1689.321058] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1689.444149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.477816] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1691.152359] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1693.260642] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1694.697776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1694.710936] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1694.728781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1699.298101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1699.302964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.309477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1699.813785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1701.938101] LustreError: 68206:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1701.940632] LustreError: 68206:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1701.987122] Lustre: server umount lustre-OST0000 complete [ 1704.929712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.931506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1704.937082] Lustre: Skipped 1 previous similar message [ 1704.938207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.760898] LustreError: 68407:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1709.763568] LustreError: 68407:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1709.786323] Lustre: server umount lustre-MDT0000 complete [ 1710.049144] LustreError: 65333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1710.054364] LustreError: 65333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1711.280571] LustreError: 65339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080585 with bad export cookie 16420471440533110525 [ 1711.282657] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1711.286029] LustreError: 65339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1711.435404] Lustre: server umount lustre-MDT0001 complete [ 1715.504207] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 1716.607026] Key type lgssc unregistered [ 1716.739944] LNet: 69139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.742970] LNetError: 69139:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1716.754338] LNet: Removed LNI 192.168.203.117@tcp [ 1717.080316] Key type .llcrypt unregistered [ 1717.081603] Key type ._llcrypt unregistered [ 1726.629019] Key type ._llcrypt registered [ 1726.630751] Key type .llcrypt registered [ 1726.678653] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1727.034772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1727.078441] alg: No test for adler32 (adler32-zlib) [ 1727.929871] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 1728.021729] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 1729.608169] Key type lgssc registered [ 1729.996716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1732.946296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1732.953739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.172485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1735.807981] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1739.427423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1739.617558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1741.104860] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1743.257661] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1744.790481] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1747.741583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1747.918375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.785988] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1753.471644] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1759.218663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1762.144502] Lustre: Failing over lustre-MDT0000 [ 1762.269565] LustreError: 72757:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1762.311738] Lustre: server umount lustre-MDT0000 complete [ 1764.321354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.324973] LustreError: 70505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1764.331169] Lustre: Skipped 2 previous similar messages [ 1764.345349] LustreError: 70505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1764.954887] LustreError: 70507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1766.716996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.803701] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1766.866339] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1766.926238] LustreError: 73264:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 1766.929114] LustreError: 73264:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.117@tcp: cfg command failed: rc = -9 [ 1766.933391] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1766.933391] [ 1766.937627] LustreError: MGC192.168.203.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1766.943618] LustreError: 73252:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1766.946254] LustreError: 73252:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 1766.949443] LustreError: 73252:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 1766.952426] LustreError: 73252:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 1766.963301] Lustre: server umount lustre-MDT0000 complete [ 1766.965392] LustreError: 73252:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1769.443470] LustreError: 70511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1769.456133] LustreError: 70511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1769.900704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.000924] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1770.064853] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.17@tcp (not set up) [ 1770.118307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1770.146129] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1771.554478] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1771.800285] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1775.593607] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1775.609767] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1775.634314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1779.681236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1779.685303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1779.689952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1780.705421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1780.708769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.709938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1780.716170] Lustre: Skipped 1 previous similar message [ 1783.160237] LustreError: 74074:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1783.166843] LustreError: 74074:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1783.218905] Lustre: server umount lustre-OST0000 complete [ 1785.825413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1785.827121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.830616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.834422] Lustre: Skipped 1 previous similar message [ 1790.945283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1790.948332] Lustre: Skipped 2 previous similar messages [ 1799.136138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1799.178732] LustreError: 74277:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1799.181080] LustreError: 74277:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1799.216766] Lustre: server umount lustre-MDT0000 complete [ 1801.105099] LustreError: 70492:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080675 with bad export cookie 2411890969336027985 [ 1801.114152] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1801.210342] LustreError: 74479:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1801.216850] LustreError: 74479:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1801.308349] Lustre: server umount lustre-MDT0001 complete [ 1805.873590] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 1807.169756] Key type lgssc unregistered [ 1807.326493] LNet: 75010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.330788] LNetError: 75010:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.337306] LNet: Removed LNI 192.168.203.117@tcp [ 1807.672114] Key type .llcrypt unregistered [ 1807.673565] Key type ._llcrypt unregistered [ 1813.031623] Key type ._llcrypt registered [ 1813.032971] Key type .llcrypt registered [ 1813.078695] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 1813.221887] Key type .llcrypt unregistered [ 1813.224034] Key type ._llcrypt unregistered [ 1823.466075] Key type ._llcrypt registered [ 1823.467423] Key type .llcrypt registered [ 1823.515395] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1823.833102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1823.848597] alg: No test for adler32 (adler32-zlib) [ 1824.722389] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 1824.827313] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 1826.416121] Key type lgssc registered [ 1826.847671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1830.061767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1830.068176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.299191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.955659] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1836.370076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1836.529482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1838.081909] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1840.328674] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1842.046691] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1845.153948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1845.331754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.784734] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1850.175103] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1856.615571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1867.234868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1867.235370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.238193] LustreError: Skipped 1 previous similar message [ 1867.243920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.247322] Lustre: Skipped 1 previous similar message [ 1871.149087] LustreError: 79121:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1871.192377] Lustre: server umount lustre-OST0000 complete [ 1877.474571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.476320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.481907] Lustre: Skipped 2 previous similar messages [ 1877.486396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1882.594392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1882.599064] Lustre: Skipped 2 previous similar messages [ 1887.200164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1887.242972] LustreError: 79322:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1887.245854] LustreError: 79322:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1887.281950] Lustre: server umount lustre-MDT0000 complete [ 1887.714043] LustreError: 76851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1887.719779] LustreError: 76851:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1889.123850] LustreError: 76838:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080763 with bad export cookie 10311364757246568820 [ 1889.128720] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1889.132665] LustreError: 76838:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1889.231075] LustreError: 79525:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1889.233478] LustreError: 79525:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1889.290500] Lustre: server umount lustre-MDT0001 complete [ 1893.611607] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 1894.906962] Key type lgssc unregistered [ 1895.036392] LNet: 80056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.041489] LNetError: 80056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.052405] LNet: Removed LNI 192.168.203.117@tcp [ 1895.406161] Key type .llcrypt unregistered [ 1895.407334] Key type ._llcrypt unregistered [ 1905.688679] Key type ._llcrypt registered [ 1905.690869] Key type .llcrypt registered [ 1905.735093] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1906.121205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1906.151079] alg: No test for adler32 (adler32-zlib) [ 1907.036685] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 1907.144097] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 1908.752968] Key type lgssc registered [ 1909.222983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1913.602400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1913.609607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.884058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.643725] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1921.250497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1921.416703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1923.381544] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1925.740513] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1927.258774] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1930.169124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1930.334471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.780621] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1935.214453] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1940.585973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1946.086932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.087969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1946.093411] Lustre: Skipped 1 previous similar message [ 1946.098557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1949.937194] LustreError: 83671:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1949.982627] Lustre: server umount lustre-OST0000 complete [ 1952.224915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1952.229046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1952.234815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1952.236547] Lustre: Skipped 1 previous similar message [ 1956.322831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1956.325750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.763283] LustreError: 83872:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1957.765740] LustreError: 83872:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1957.804163] Lustre: server umount lustre-MDT0000 complete [ 1959.545468] LustreError: 81407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080833 with bad export cookie 3295781733382208863 [ 1959.549724] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1959.550555] LustreError: 81407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1959.622692] LustreError: 84073:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1959.625184] LustreError: 84073:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1959.700409] Lustre: server umount lustre-MDT0001 complete [ 1964.433841] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 1965.656266] Key type lgssc unregistered [ 1965.809443] LNet: 84605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.812371] LNetError: 84605:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1965.824814] LNet: Removed LNI 192.168.203.117@tcp [ 1966.209034] Key type .llcrypt unregistered [ 1966.210375] Key type ._llcrypt unregistered [ 1976.765077] Key type ._llcrypt registered [ 1976.766507] Key type .llcrypt registered [ 1976.814490] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1977.271533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1977.287397] alg: No test for adler32 (adler32-zlib) [ 1978.159264] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 1978.248735] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 1979.848184] Key type lgssc registered [ 1980.319046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1983.613789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1983.625276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1984.909463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.729367] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1990.296711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1990.426062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.875020] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1994.056453] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1995.637846] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1998.466679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1998.620606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2001.146862] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2003.775322] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2009.899841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2020.322768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.323897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2020.324287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2020.327072] Lustre: Skipped 1 previous similar message [ 2024.113086] LustreError: 88241:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2024.170459] Lustre: server umount lustre-OST0000 complete [ 2026.465222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2026.468790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.475416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.478093] Lustre: Skipped 1 previous similar message [ 2030.561870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2030.562153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.681139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2035.684033] Lustre: Skipped 2 previous similar messages [ 2040.288300] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2040.324690] LustreError: 88443:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2040.327596] LustreError: 88443:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2040.359361] Lustre: server umount lustre-MDT0000 complete [ 2040.804349] LustreError: 85977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2040.817560] LustreError: 85977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2041.894374] LustreError: 85956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080916 with bad export cookie 9867867536469793052 [ 2041.896293] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2041.901661] LustreError: 85956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2041.982434] LustreError: 88647:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2041.989270] LustreError: 88647:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2042.059122] Lustre: server umount lustre-MDT0001 complete [ 2046.435936] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 2047.631395] Key type lgssc unregistered [ 2047.801278] LNet: 89179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2047.805164] LNetError: 89179:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2047.821133] LNet: Removed LNI 192.168.203.117@tcp [ 2048.240391] Key type .llcrypt unregistered [ 2048.241433] Key type ._llcrypt unregistered [ 2058.484510] Key type ._llcrypt registered [ 2058.486371] Key type .llcrypt registered [ 2058.532689] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 2058.952835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2058.977865] alg: No test for adler32 (adler32-zlib) [ 2059.841368] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 2059.933602] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 2061.520198] Key type lgssc registered [ 2061.915497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2064.976191] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2064.988750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2080.480387] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2085.600202] LustreError: 90523:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.117@tcp: failed processing log, type 4: rc = -110 [ 2112.224259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.291333] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2116.353743] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2118.262920] LustreError: 91380:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2118.291591] Lustre: server umount lustre-OST0000 complete [ 2121.296576] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 06:23:15 (1769080995) [ 2123.935769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2124.096581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2125.511611] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2129.127920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2129.252723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2130.674378] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2132.677316] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2134.799115] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2137.429078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2137.582980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.667083] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2141.600713] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2148.841932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2159.076305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.078816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2159.079358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2159.081231] Lustre: Skipped 1 previous similar message [ 2161.649168] LustreError: 94472:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2161.652292] LustreError: 94472:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2161.706809] Lustre: server umount lustre-OST0000 complete [ 2164.194804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.196383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.200151] Lustre: Skipped 1 previous similar message [ 2164.204958] Lustre: Skipped 2 previous similar messages [ 2169.313142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.315104] Lustre: Skipped 1 previous similar message [ 2169.470697] LustreError: 94674:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2169.473064] LustreError: 94674:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2169.505191] Lustre: server umount lustre-MDT0000 complete [ 2170.954773] LustreError: 92031:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081045 with bad export cookie 9806136146564746202 [ 2170.956364] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2170.959460] LustreError: 92031:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2171.074711] Lustre: server umount lustre-MDT0001 complete [ 2175.075521] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 2176.223684] Key type lgssc unregistered [ 2176.361611] LNet: 95407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2176.364531] LNetError: 95407:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2176.378486] LNet: Removed LNI 192.168.203.117@tcp [ 2176.748991] Key type .llcrypt unregistered [ 2176.750501] Key type ._llcrypt unregistered [ 2196.480617] Key type ._llcrypt registered [ 2196.483631] Key type .llcrypt registered [ 2196.643074] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 2197.989303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2198.056696] alg: No test for adler32 (adler32-zlib) [ 2199.068798] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 2199.346050] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 2201.016166] Key type lgssc registered [ 2201.616333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2208.015414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2208.056128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.461794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.709734] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2219.054839] LustreError: 96799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2219.088254] LustreError: 96799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2224.101021] LustreError: 96799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2226.992712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2227.856877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2234.219096] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2243.128828] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2247.494313] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2254.572318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2254.942319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.853853] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2259.948982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2264.751437] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2275.297785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2275.305845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.324292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2277.350808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.357395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2281.349118] LustreError: 99049:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2281.536646] Lustre: server umount lustre-OST0000 complete [ 2287.588570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.591511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2287.592523] Lustre: Skipped 1 previous similar message [ 2287.610151] Lustre: Skipped 1 previous similar message [ 2290.333785] LustreError: 99252:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2290.336507] LustreError: 99252:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2292.395281] LustreError: 99252:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 2292.397477] LustreError: 99252:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2292.520345] Lustre: server umount lustre-MDT0000 complete [ 2292.717309] LustreError: 97513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.736448] LustreError: 97513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2295.540772] LustreError: 99050:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081169 with bad export cookie 16558670594629790558 [ 2295.542844] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2295.548647] LustreError: 99050:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2295.678563] LustreError: 99454:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2295.684337] LustreError: 99454:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2295.808725] Lustre: server umount lustre-MDT0001 complete [ 2303.597983] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 2305.623880] Key type lgssc unregistered [ 2305.892092] LNet: 99987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.902688] LNetError: 99987:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2306.920933] LNet: Removed LNI 192.168.203.117@tcp [ 2307.492380] Key type .llcrypt unregistered [ 2307.494374] Key type ._llcrypt unregistered [ 2320.553258] Key type ._llcrypt registered [ 2320.555728] Key type .llcrypt registered [ 2320.634870] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 2321.578399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2321.617930] alg: No test for adler32 (adler32-zlib) [ 2322.737231] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 2322.969759] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 2324.672171] Key type lgssc registered [ 2325.609211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2332.097381] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2332.111751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2347.617622] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2352.736860] LustreError: 101187:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.117@tcp: failed processing log, type 4: rc = -110 [ 2379.424181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.228352] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2388.745954] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2394.453987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.703400] LustreError: MGC192.168.203.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2394.715137] LustreError: 102099:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2394.723450] LustreError: 102099:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2394.732816] LustreError: 102099:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2394.803465] Lustre: server umount lustre-MDT0000 complete [ 2394.805426] LustreError: 102099:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2410.467767] LustreError: 102529:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2410.595790] Lustre: server umount lustre-OST0000 complete [ 2418.559695] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 2426.302440] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 2433.184795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2438.467603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2443.742937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2449.460760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2455.163633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2455.214242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2455.377838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2455.402621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2455.497836] Lustre: lustre-MDT0000: new disk, initializing [ 2455.586561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2455.617268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2459.528482] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2468.631548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2468.705074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2468.792655] Lustre: 104868:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2468.845295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2468.848666] Lustre: Skipped 1 previous similar message [ 2468.909498] Lustre: lustre-MDT0001: new disk, initializing [ 2468.987120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2469.015577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2469.025665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2472.707631] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2478.383178] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2480.931994] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2485.873397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2485.937065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2486.104346] Lustre: lustre-OST0000: new disk, initializing [ 2486.107243] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2486.164480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2490.647567] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2494.498327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2494.506597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2494.577820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2496.692612] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2499.819193] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2499.958909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2502.816872] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2502.954896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2504.677660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2504.678590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.679403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2504.688058] Lustre: Skipped 1 previous similar message [ 2509.805142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2509.812063] Lustre: Skipped 2 previous similar messages [ 2510.344775] LustreError: 107807:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2510.353604] LustreError: 107807:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2510.463906] Lustre: server umount lustre-OST0000 complete [ 2514.915463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2514.916095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2514.918469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2514.922992] LustreError: Skipped 1 previous similar message [ 2514.941221] Lustre: Skipped 1 previous similar message [ 2519.420068] LustreError: 108008:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2519.426348] LustreError: 108008:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2519.546353] Lustre: server umount lustre-MDT0000 complete [ 2522.082478] LustreError: 104860:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081396 with bad export cookie 15570317885187999517 [ 2522.086548] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2522.087419] LustreError: 104860:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2522.181640] LustreError: 108210:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2522.186092] LustreError: 108210:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2522.299881] Lustre: server umount lustre-MDT0001 complete [ 2528.680138] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 06:30:01 (1769081401) [ 2531.028555] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2544.870312] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 2552.224662] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 2561.068807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2569.406431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2577.722845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2582.485974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2588.941118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2589.010554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.202691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2589.255173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2589.342729] Lustre: lustre-MDT0000: new disk, initializing [ 2589.426179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.448811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2592.807726] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2601.873305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2601.962221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2602.059416] Lustre: 111346:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2602.120984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2602.124571] Lustre: Skipped 1 previous similar message [ 2602.241599] Lustre: lustre-MDT0001: new disk, initializing [ 2602.317679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2602.343675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2602.362936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2606.333232] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2612.647793] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2615.875284] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2621.332495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2621.405623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2621.640559] Lustre: lustre-OST0000: new disk, initializing [ 2621.646442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2621.727530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2626.178575] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2631.225179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2631.236745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2631.303851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2632.308508] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2635.928162] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2636.081260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2639.272945] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2639.422055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2641.384122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.385594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.410265] Lustre: Skipped 1 previous similar message [ 2641.435527] Lustre: Skipped 2 previous similar messages [ 2646.505479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2646.513389] Lustre: Skipped 1 previous similar message [ 2655.200184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2655.235500] LustreError: 114282:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2655.243849] LustreError: 114282:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2655.333826] Lustre: server umount lustre-OST0000 complete [ 2658.788297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2658.792385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.811447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2658.816869] Lustre: Skipped 2 previous similar messages [ 2664.419638] LustreError: 114486:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2664.423569] LustreError: 114486:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2664.545782] Lustre: server umount lustre-MDT0000 complete [ 2666.983196] LustreError: 111359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.993408] LustreError: 111359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2667.481101] LustreError: 111339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081541 with bad export cookie 15570317885188000392 [ 2667.493303] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2667.494068] LustreError: 111339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2667.745657] Lustre: server umount lustre-MDT0001 complete [ 2674.647444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.999133] LustreError: 115164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.065803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2678.532115] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2680.296144] LustreError: 115164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.311034] LustreError: 115164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2685.422957] LustreError: 115164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.553508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2688.642473] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2692.285066] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2694.501796] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2698.866333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2703.119288] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2706.945352] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2715.673914] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2719.717475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2719.729227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.738828] Lustre: Skipped 1 previous similar message [ 2719.743697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2719.746404] Lustre: Skipped 2 previous similar messages [ 2724.021176] LustreError: 117504:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2724.024205] LustreError: 117504:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2724.105631] Lustre: server umount lustre-OST0000 complete [ 2726.886209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2726.889628] LustreError: Skipped 1 previous similar message [ 2730.978459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2730.983988] Lustre: Skipped 2 previous similar messages [ 2732.299712] Lustre: server umount lustre-MDT0000 complete [ 2734.194539] LustreError: 115145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081608 with bad export cookie 15570317885188001281 [ 2734.195988] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2734.200205] LustreError: 115145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2734.359340] Lustre: server umount lustre-MDT0001 complete [ 2738.856738] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 2740.198772] Key type lgssc unregistered [ 2740.359424] LNet: 118439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2740.362291] LNetError: 118439:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2740.372320] LNet: Removed LNI 192.168.203.117@tcp [ 2740.741806] Key type .llcrypt unregistered [ 2740.743716] Key type ._llcrypt unregistered [ 2748.617758] Key type ._llcrypt registered [ 2748.618987] Key type .llcrypt registered [ 2748.669109] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 2755.650519] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 2756.194677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2756.205028] alg: No test for adler32 (adler32-zlib) [ 2757.107266] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 2757.216718] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 2758.824453] Key type lgssc registered [ 2759.353332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2762.764047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2765.360937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2767.811015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2770.175744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2772.902598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2772.936483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2772.944119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.089576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2774.107466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2774.152433] Lustre: lustre-MDT0000: new disk, initializing [ 2774.187827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2774.195570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2775.676718] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2780.114488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2780.133804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2780.170510] Lustre: 121541:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2780.199467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2780.204464] Lustre: Skipped 1 previous similar message [ 2780.242980] Lustre: lustre-MDT0001: new disk, initializing [ 2780.276858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2780.289861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2780.293163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2781.707688] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2784.935156] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2786.259657] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2789.215888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2789.250841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2789.396273] Lustre: lustre-OST0000: new disk, initializing [ 2789.398402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2789.434208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.690833] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2795.133553] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2796.030791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2796.036899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2796.082219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2796.730384] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2796.811067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2798.297295] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2798.370818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2801.123388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2801.124853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2801.128208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2801.132262] LustreError: Skipped 1 previous similar message [ 2801.137228] Lustre: Skipped 1 previous similar message [ 2805.235154] LustreError: 124478:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2805.289051] Lustre: server umount lustre-OST0000 complete [ 2811.361408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2811.362134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2811.363249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2811.367211] Lustre: Skipped 2 previous similar messages [ 2816.481668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2816.483900] Lustre: Skipped 2 previous similar messages [ 2821.088213] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2821.127640] LustreError: 124680:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2821.130063] LustreError: 124680:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2821.161857] Lustre: server umount lustre-MDT0000 complete [ 2822.580347] LustreError: 122322:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081696 with bad export cookie 7586167895912725261 [ 2822.583871] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2822.590273] LustreError: 122322:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2822.666613] LustreError: 124885:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2822.669502] LustreError: 124885:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2822.734690] Lustre: server umount lustre-MDT0001 complete [ 2826.113859] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 06:34:59 (1769081699) [ 2828.645180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2828.805476] LustreError: 125557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.814243] LustreError: 125557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2828.834671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2830.479240] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2833.889117] LustreError: 125557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.983428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2834.117259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2835.574911] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2837.603615] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2838.879273] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2843.617581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.618136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2843.622679] Lustre: Skipped 1 previous similar message [ 2845.693925] LustreError: 126898:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2845.695762] LustreError: 126898:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2845.731190] Lustre: server umount lustre-MDT0000 complete [ 2847.333486] LustreError: 125536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081721 with bad export cookie 7586167895912726129 [ 2847.336873] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2847.338322] LustreError: 125536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2847.470964] Lustre: server umount lustre-MDT0001 complete [ 2851.633938] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 06:35:25 (1769081725) [ 2854.603452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2870.048548] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2875.168455] LustreError: 127743:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.117@tcp: failed processing log, type 4: rc = -110 [ 2900.768642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2903.028430] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2905.209083] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2906.050945] LustreError: 128407:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2906.053509] LustreError: 128407:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2906.101234] Lustre: server umount lustre-OST0000 complete [ 2909.776454] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 06:36:23 (1769081783) [ 2912.752724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2913.001739] LustreError: 129077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.874664] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2918.314093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2918.369771] LustreError: 129076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.376481] LustreError: 129076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2918.482552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2918.485731] Lustre: Skipped 1 previous similar message [ 2920.022549] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2922.337308] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2924.298441] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2927.200574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2929.364202] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2931.511525] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2943.972621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2943.975375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.980149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2943.982919] Lustre: Skipped 3 previous similar messages [ 2949.957251] LustreError: 131342:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2949.961893] LustreError: 131342:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2949.994636] Lustre: server umount lustre-MDT0000 complete [ 2950.114374] LustreError: 129783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.121957] LustreError: 129783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2951.661780] LustreError: 129056:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081825 with bad export cookie 7586167895912726780 [ 2951.666354] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2951.794915] Lustre: server umount lustre-MDT0001 complete [ 2963.483512] Lustre: server umount lustre-OST0000 complete [ 2966.487749] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 06:37:20 (1769081840) [ 2969.012874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2969.160395] LustreError: 132417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.185370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2969.187407] Lustre: Skipped 1 previous similar message [ 2970.530537] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2973.519039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2975.174402] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2977.223179] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2978.610821] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2981.005756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2983.026918] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2984.825307] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2986.217413] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2994.472942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2995.466226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2997.065084] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2997.167961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2998.753420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2998.757417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.762311] Lustre: Skipped 2 previous similar messages [ 2998.764813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2998.767014] Lustre: Skipped 3 previous similar messages [ 3000.049145] LustreError: 135223:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3000.053216] LustreError: 135223:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3000.101507] Lustre: server umount lustre-OST0000 complete [ 3004.384965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3004.388231] LustreError: Skipped 1 previous similar message [ 3007.705637] Lustre: server umount lustre-MDT0000 complete [ 3009.077169] LustreError: 132397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081883 with bad export cookie 7586167895912727837 [ 3009.081332] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3009.082209] LustreError: 132397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3009.202913] Lustre: server umount lustre-MDT0001 complete [ 3012.293518] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 06:38:06 (1769081886) [ 3014.725979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3030.176574] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3035.296476] LustreError: 136267:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.117@tcp: failed processing log, type 4: rc = -110 [ 3060.896351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.899345] Lustre: Skipped 2 previous similar messages [ 3062.955610] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3064.961467] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3067.384152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3067.524961] LustreError: 136294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.569099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3068.849854] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3071.853678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3073.434356] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3075.482502] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3076.806076] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3078.250474] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3078.347502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3079.812060] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3079.898018] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3082.211312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3082.214178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.219080] Lustre: Skipped 3 previous similar messages [ 3082.221399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3082.223755] Lustre: Skipped 4 previous similar messages [ 3086.786621] LustreError: 139046:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3086.789598] LustreError: 139046:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3086.818637] Lustre: server umount lustre-MDT0000 complete [ 3088.184725] LustreError: 136276:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081962 with bad export cookie 7586167895912728845 [ 3088.190021] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3088.191910] LustreError: 136276:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3088.295743] Lustre: server umount lustre-MDT0001 complete [ 3098.965239] Lustre: server umount lustre-OST0000 complete [ 3101.664140] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 06:39:35 (1769081975) [ 3103.906894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3119.392735] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3124.512430] LustreError: 140093:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.117@tcp: failed processing log, type 4: rc = -110 [ 3152.251671] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3154.232233] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3156.834140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3156.982467] LustreError: 140120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3157.027326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3158.281712] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3161.227887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3162.817593] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3164.722183] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3165.955580] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3168.252058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3168.283205] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3168.343646] Lustre: lustre-OST0001: new disk, initializing [ 3168.346066] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3170.421516] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3170.492630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3170.496505] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3170.553145] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3173.756572] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3175.179497] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3175.264948] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3176.853116] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3176.940948] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3181.027215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3181.030938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3181.031337] Lustre: Skipped 3 previous similar messages [ 3181.033318] Lustre: Skipped 3 previous similar messages [ 3192.288134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3192.303174] LustreError: 143822:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3192.304923] LustreError: 143822:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3192.320811] Lustre: server umount lustre-OST0000 complete [ 3196.385867] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3196.389047] LustreError: Skipped 1 previous similar message [ 3199.757374] Lustre: server umount lustre-OST0001 complete [ 3215.328093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3215.385904] Lustre: server umount lustre-MDT0000 complete [ 3216.768264] LustreError: 142628:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082090 with bad export cookie 7586167895912729790 [ 3216.771663] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3216.773690] LustreError: 142628:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3216.916216] Lustre: server umount lustre-MDT0001 complete [ 3219.106154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3220.687126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3222.189064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3223.617839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3226.085536] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 06:41:39 (1769082099) [ 3226.654778] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3227.214723] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3227.855501] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 06:41:41 (1769082101) [ 3230.310033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3230.401887] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3230.411960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3230.480764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3230.483464] Lustre: Skipped 6 previous similar messages [ 3231.824813] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3234.983280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3235.022167] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3235.033365] Lustre: 146060:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3236.584272] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3238.528893] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3239.812291] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3242.108570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3242.190131] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3244.270566] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3245.032818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3246.216252] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3247.709635] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3247.789167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3249.099244] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3249.170719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3250.146870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3250.147227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.149795] LustreError: Skipped 1 previous similar message [ 3250.154072] Lustre: Skipped 4 previous similar messages [ 3250.154688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3250.158603] Lustre: Skipped 14 previous similar messages [ 3251.926738] Lustre: server umount lustre-OST0000 complete [ 3265.843193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3267.856692] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3269.668221] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3270.056697] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3270.061569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3270.061810] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3270.069071] Lustre: Skipped 1 previous similar message [ 3271.205569] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3271.276668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3272.705806] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3272.789417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3274.064600] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f91d034e000.ost_server_uuid 50 [ 3274.592177] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f91d034e000.ost_server_uuid in FULL state after 0 sec [ 3274.649386] LustreError: 146068:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.203.17@tcp) returned error from blocking AST (req@000000003205c500 x1855016540500352 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000f1ac7920/0x69477b821259d603 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.17@tcp remote: 0x16b5f6a704d0cad1 expref: 5 pid: 147495 timeout: 3374 lvb_type: 0 [ 3274.661138] LustreError: lustre-MDT0000: A client on nid 192.168.203.17@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3274.664614] LustreError: 146057:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.17@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000f1ac7920/0x69477b821259d603 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.17@tcp remote: 0x16b5f6a704d0cad1 expref: 6 pid: 147495 timeout: 0 lvb_type: 0 [ 3281.874674] Lustre: server umount lustre-OST0000 complete [ 3289.372887] Lustre: server umount lustre-MDT0000 complete [ 3290.594172] LustreError: 146072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3290.602756] LustreError: 146072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3290.860319] LustreError: 146052:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082165 with bad export cookie 7586167895912730966 [ 3290.864225] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3290.864908] LustreError: 146052:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3297.014857] Lustre: server umount lustre-MDT0001 complete [ 3300.768650] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 3301.834887] Key type lgssc unregistered [ 3301.962351] LNet: 151542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3301.967392] LNetError: 151542:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3301.976344] LNet: Removed LNI 192.168.203.117@tcp [ 3302.274426] Key type .llcrypt unregistered [ 3302.276149] Key type ._llcrypt unregistered [ 3311.590413] Key type ._llcrypt registered [ 3311.591766] Key type .llcrypt registered [ 3311.638997] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3312.012844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3312.057072] alg: No test for adler32 (adler32-zlib) [ 3312.919436] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 3313.005878] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 3314.584139] Key type lgssc registered [ 3315.034412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3317.852282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3317.860366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.024701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3320.442841] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3323.572965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3323.713235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3325.101118] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3326.964772] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3328.188422] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3330.578184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3330.718384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3332.708498] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3334.422553] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3344.043801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3347.324240] Lustre: Failing over lustre-MDT0000 [ 3347.506282] LustreError: 155175:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3347.535104] Lustre: server umount lustre-MDT0000 complete [ 3349.475397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.475424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3349.475956] LustreError: 152908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3349.475963] LustreError: 152908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3349.482738] Lustre: Skipped 1 previous similar message [ 3354.595158] LustreError: 152913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.601669] LustreError: 152913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3358.550210] LustreError: 152894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.17@tcp arrived at 1769082232 with bad export cookie 869559051440547673 [ 3358.555710] LustreError: 152894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3359.715532] LustreError: 152913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.720570] LustreError: 152913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3361.400835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.444572] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3361.546318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3361.567101] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3362.921513] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3363.708687] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3363.713028] Lustre: lustre-MDT0000: Denying connection for new client 29595ae7-d274-4c58-87b3-feb1efb24cc3 (at 192.168.203.17@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3366.885395] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3372.004738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3372.005652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3372.009330] Lustre: Skipped 2 previous similar messages [ 3372.014205] Lustre: Skipped 1 previous similar message [ 3376.559100] LustreError: 156108:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3376.561166] LustreError: 156108:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3376.605719] Lustre: server umount lustre-OST0000 complete [ 3377.910508] LustreError: 156310:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3377.913700] Lustre: 155676:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3377.916666] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3377.923501] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3377.928955] LustreError: 155676:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c97b6bbc x1855017123351168/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3377.937737] LustreError: 155676:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3377.941457] LustreError: 155676:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3377.945070] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 3378.026061] LustreError: 156310:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3378.028325] LustreError: 156310:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3378.055667] Lustre: server umount lustre-MDT0000 complete [ 3379.321359] LustreError: 152892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082253 with bad export cookie 869559051440548023 [ 3379.325251] LustreError: 152222:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3379.326243] LustreError: 152892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3379.326296] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3385.470443] LustreError: 156512:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3385.472427] LustreError: 156512:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3385.529630] Lustre: server umount lustre-MDT0001 complete [ 3389.400333] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 3390.560250] Key type lgssc unregistered [ 3390.717597] LNet: 157045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3390.723304] LNetError: 157045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3390.731654] LNet: Removed LNI 192.168.203.117@tcp [ 3391.046107] Key type .llcrypt unregistered [ 3391.047612] Key type ._llcrypt unregistered [ 3400.254322] Key type ._llcrypt registered [ 3400.255845] Key type .llcrypt registered [ 3400.305406] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3400.821650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3400.841322] alg: No test for adler32 (adler32-zlib) [ 3401.707818] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 3401.796751] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 3403.395060] Key type lgssc registered [ 3403.784709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3406.548328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3406.556322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.744143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3409.152494] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3412.283248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3412.424284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3413.950285] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3415.808460] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3416.972839] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3419.343665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3419.469836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.413216] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3423.266758] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3432.810373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3438.052403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.053819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3438.057305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3438.059451] LustreError: Skipped 1 previous similar message [ 3440.884109] LustreError: 160681:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3440.923222] Lustre: server umount lustre-OST0000 complete [ 3443.170855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3443.171069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3443.171576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3443.171580] Lustre: Skipped 1 previous similar message [ 3443.176811] Lustre: Skipped 2 previous similar messages [ 3448.289408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3448.292196] Lustre: Skipped 2 previous similar messages [ 3448.382508] LustreError: 160883:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3448.384761] LustreError: 160883:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3448.409450] Lustre: server umount lustre-MDT0000 complete [ 3449.766737] LustreError: 158396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082323 with bad export cookie 9334197561697817528 [ 3449.769829] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3449.771408] LustreError: 158396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3449.843767] LustreError: 161085:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3449.846742] LustreError: 161085:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3449.905874] Lustre: server umount lustre-MDT0001 complete [ 3453.675457] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 3454.801435] Key type lgssc unregistered [ 3454.922504] LNet: 161616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3454.925269] LNetError: 161616:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3454.934295] LNet: Removed LNI 192.168.203.117@tcp [ 3455.246411] Key type .llcrypt unregistered [ 3455.247779] Key type ._llcrypt unregistered [ 3467.248616] Key type ._llcrypt registered [ 3467.249697] Key type .llcrypt registered [ 3467.294639] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3467.707164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3467.732298] alg: No test for adler32 (adler32-zlib) [ 3468.621218] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 3468.720528] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 3470.312447] Key type lgssc registered [ 3470.716647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3473.733593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3473.742833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.945121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3476.589361] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3479.871619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3480.039256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3481.550741] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3483.461485] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3484.735272] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3487.306105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3487.454427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3489.493159] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3491.466229] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3500.776441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3506.149482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3506.150377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3506.151661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3506.153703] LustreError: Skipped 1 previous similar message [ 3506.158365] Lustre: Skipped 1 previous similar message [ 3509.872138] LustreError: 165543:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3509.913765] Lustre: server umount lustre-OST0000 complete [ 3515.873571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3515.876682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.881777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3515.883949] Lustre: Skipped 1 previous similar message [ 3517.443870] LustreError: 165745:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3517.446067] LustreError: 165745:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3517.480936] Lustre: server umount lustre-MDT0000 complete [ 3518.848690] LustreError: 164893:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082393 with bad export cookie 17317682864378110640 [ 3518.849674] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3518.853223] LustreError: 164893:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3518.924974] LustreError: 165947:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3518.927130] LustreError: 165947:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3518.980779] Lustre: server umount lustre-MDT0001 complete [ 3522.725585] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 3523.813457] Key type lgssc unregistered [ 3523.953478] LNet: 166477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3523.956221] LNetError: 166477:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3523.964370] LNet: Removed LNI 192.168.203.117@tcp [ 3524.288054] Key type .llcrypt unregistered [ 3524.289238] Key type ._llcrypt unregistered [ 3532.820875] Key type ._llcrypt registered [ 3532.822405] Key type .llcrypt registered [ 3532.863975] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3533.354693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3533.363089] alg: No test for adler32 (adler32-zlib) [ 3534.227637] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 3534.344503] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 3535.936187] Key type lgssc registered [ 3536.347694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3539.928443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3539.934569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3541.081736] Lustre: *** cfs_fail_loc=135, val=0*** [ 3541.127584] LustreError: 167973:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 3541.130508] LustreError: 167973:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.117@tcp: cfg command failed: rc = -2 [ 3541.133492] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3541.133492] [ 3541.136163] LustreError: MGC192.168.203.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3541.141376] LustreError: 167933:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3541.144479] LustreError: 167933:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 3541.147066] LustreError: 167933:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 3541.149398] LustreError: 167933:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 3541.152352] LustreError: 167933:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3541.184361] Lustre: server umount lustre-MDT0000 complete [ 3541.185551] LustreError: 167933:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3543.513974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3543.680533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3544.992969] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3547.898388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3548.029933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3549.348268] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3551.029886] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3552.209070] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3552.894390] LustreError: 168375:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3552.896027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3552.896658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3552.897618] LustreError: 168375:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3552.901646] Lustre: Skipped 1 previous similar message [ 3552.906921] LustreError: 168375:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 3558.049281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3558.051094] Lustre: Skipped 2 previous similar messages [ 3559.036035] LustreError: 169688:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3559.038048] LustreError: 169688:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3559.060423] Lustre: server umount lustre-MDT0000 complete [ 3560.281336] LustreError: 168327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082434 with bad export cookie 14981982021415814182 [ 3560.284169] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3560.285323] LustreError: 168327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3560.344825] LustreError: 169891:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3560.347150] LustreError: 169891:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3560.393470] Lustre: server umount lustre-MDT0001 complete [ 3563.917566] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 3564.924780] Key type lgssc unregistered [ 3565.049399] LNet: 170422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3565.051588] LNetError: 170422:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3565.062343] LNet: Removed LNI 192.168.203.117@tcp [ 3565.357400] Key type .llcrypt unregistered [ 3565.358340] Key type ._llcrypt unregistered [ 3572.399847] Key type ._llcrypt registered [ 3572.400899] Key type .llcrypt registered [ 3572.442632] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 3572.544574] Key type .llcrypt unregistered [ 3572.546240] Key type ._llcrypt unregistered [ 3579.779569] Key type ._llcrypt registered [ 3579.780961] Key type .llcrypt registered [ 3579.819754] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3580.312466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3580.334106] alg: No test for adler32 (adler32-zlib) [ 3581.208895] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 3581.297330] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 3582.888157] Key type lgssc registered [ 3583.313187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3586.375368] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3586.384259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3601.824298] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3606.944189] LustreError: 172491:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.117@tcp: failed processing log, type 4: rc = -110 [ 3633.568172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3635.559911] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3637.415783] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3639.799948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3639.993444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3640.026157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3641.456773] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3644.649734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3644.839273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3646.316656] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3648.202659] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3649.399147] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3650.408139] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3655.649159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3655.651739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.656295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3658.211065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3658.211896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3661.363145] LustreError: 175121:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3661.407957] Lustre: server umount lustre-OST0000 complete [ 3663.329731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.330352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3663.335919] Lustre: Skipped 1 previous similar message [ 3663.338320] Lustre: Skipped 1 previous similar message [ 3668.448987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3668.451209] Lustre: Skipped 2 previous similar messages [ 3673.569087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3673.571575] Lustre: Skipped 1 previous similar message [ 3677.152116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3677.183050] LustreError: 175322:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3677.185053] LustreError: 175322:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3677.213133] Lustre: server umount lustre-MDT0000 complete [ 3678.456242] LustreError: 172500:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082552 with bad export cookie 18364087988881514198 [ 3678.459240] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3678.462486] LustreError: 172500:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3678.523737] LustreError: 175525:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3678.525809] LustreError: 175525:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3678.576909] Lustre: server umount lustre-MDT0001 complete [ 3682.265354] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 3683.321575] Key type lgssc unregistered [ 3683.464309] LNet: 176056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3683.466418] LNetError: 176056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3683.480347] LNet: Removed LNI 192.168.203.117@tcp [ 3683.791562] Key type .llcrypt unregistered [ 3683.792874] Key type ._llcrypt unregistered [ 3693.091525] Key type ._llcrypt registered [ 3693.093112] Key type .llcrypt registered [ 3693.128245] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3693.525549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3693.534407] alg: No test for adler32 (adler32-zlib) [ 3694.388325] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 3694.467331] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 3696.048138] Key type lgssc registered [ 3696.427688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3699.132596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3699.143026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3700.312099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3701.774326] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3704.839112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3704.992702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3706.446160] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3708.434295] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3709.750994] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3712.074940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3712.186908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.076939] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3715.880876] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3725.542907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3727.526800] Lustre: Failing over lustre-MDT0000 [ 3727.589076] LustreError: 179745:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3727.619205] Lustre: server umount lustre-MDT0000 complete [ 3730.914476] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.915319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3730.916834] LustreError: 178702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3730.916845] LustreError: 178702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3730.917858] Lustre: Skipped 2 previous similar messages [ 3731.535649] LustreError: 177424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3731.542601] LustreError: 177424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3736.033488] LustreError: 177428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3736.039056] LustreError: 177428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3740.380903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3740.416980] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3740.498346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3740.527314] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3741.678680] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3741.768905] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3743.048946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3745.765141] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3745.776189] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3745.791211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3748.733575] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3750.882615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3750.882701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.884468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3750.890428] Lustre: Skipped 1 previous similar message [ 3756.004378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3756.005991] Lustre: Skipped 2 previous similar messages [ 3759.012136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3759.014113] Lustre: Skipped 1 previous similar message [ 3764.196624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3764.199298] Lustre: Skipped 1 previous similar message [ 3765.216192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3765.236107] LustreError: 181071:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3765.239247] LustreError: 181071:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3765.277467] Lustre: server umount lustre-OST0000 complete [ 3769.313352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3769.315802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3769.317239] Lustre: Skipped 1 previous similar message [ 3772.672731] LustreError: 181277:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3772.675233] LustreError: 181277:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3772.708642] Lustre: server umount lustre-MDT0000 complete [ 3774.005836] LustreError: 177409:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082648 with bad export cookie 17353327625029805991 [ 3774.008661] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3774.011273] LustreError: 177409:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3774.126388] Lustre: server umount lustre-MDT0001 complete [ 3778.017153] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 3779.014829] Key type lgssc unregistered [ 3779.141396] LNet: 182010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3779.143603] LNetError: 182010:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3779.152295] LNet: Removed LNI 192.168.203.117@tcp [ 3779.450496] Key type .llcrypt unregistered [ 3779.451671] Key type ._llcrypt unregistered [ 3788.461723] Key type ._llcrypt registered [ 3788.463057] Key type .llcrypt registered [ 3788.500714] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3788.828671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3788.860654] alg: No test for adler32 (adler32-zlib) [ 3789.715478] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 3789.796414] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 3791.376096] Key type lgssc registered [ 3791.722359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3794.272658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3794.279325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3795.462519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.805197] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3799.564380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3799.689806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3801.010686] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3802.731334] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3803.878930] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3806.037946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3806.158121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3807.916263] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3809.587985] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3820.522055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3825.784186] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3836.607115] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3842.516673] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3850.208789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3850.210605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.214642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3851.235318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3851.235618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3851.239686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3851.241574] Lustre: Skipped 1 previous similar message [ 3855.599103] LustreError: 185852:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3855.638607] Lustre: server umount lustre-OST0000 complete [ 3861.474494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3861.474792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3861.475773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3861.484748] Lustre: Skipped 1 previous similar message [ 3866.593226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3866.595412] Lustre: Skipped 2 previous similar messages [ 3871.200102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3871.235732] LustreError: 186060:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3871.238873] LustreError: 186060:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3871.265843] Lustre: server umount lustre-MDT0000 complete [ 3871.713182] LustreError: 183393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3871.719568] LustreError: 183393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3872.599882] LustreError: 185620:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082746 with bad export cookie 11105332853603732169 [ 3872.601582] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3872.603093] LustreError: 185620:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3872.666570] LustreError: 186263:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3872.669246] LustreError: 186263:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3872.716735] Lustre: server umount lustre-MDT0001 complete [ 3875.974733] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 3876.889671] Key type lgssc unregistered [ 3877.020367] LNet: 186794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3877.023516] LNetError: 186794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3877.031372] LNet: Removed LNI 192.168.203.117@tcp [ 3877.322520] Key type .llcrypt unregistered [ 3877.323534] Key type ._llcrypt unregistered [ 3885.744971] Key type ._llcrypt registered [ 3885.746068] Key type .llcrypt registered [ 3885.787438] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3886.143057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3886.183136] alg: No test for adler32 (adler32-zlib) [ 3887.026158] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 3887.099594] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 3888.672107] Key type lgssc registered [ 3889.025498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3891.605425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3891.613077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.774170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3894.099207] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3896.992828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3897.124379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3898.457975] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3900.293826] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3901.509590] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3903.797337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3903.932878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3905.881430] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3907.637868] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3917.223391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3919.405058] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3929.713979] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3936.459966] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3944.212571] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3956.195121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3956.196204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3956.199213] Lustre: Skipped 1 previous similar message [ 3959.343103] LustreError: 192398:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3959.382358] Lustre: server umount lustre-OST0000 complete [ 3961.313807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.317134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3961.320340] Lustre: Skipped 1 previous similar message [ 3961.323588] Lustre: Skipped 1 previous similar message [ 3966.433028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3966.434888] Lustre: Skipped 1 previous similar message [ 3966.784751] LustreError: 192599:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3966.786428] LustreError: 192599:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3966.813145] Lustre: server umount lustre-MDT0000 complete [ 3968.078458] LustreError: 188145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082842 with bad export cookie 441320625670137041 [ 3968.082496] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3968.083320] LustreError: 188145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3968.162818] LustreError: 192800:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3968.164889] LustreError: 192800:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3968.223801] Lustre: server umount lustre-MDT0001 complete [ 3971.949355] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 3973.013207] Key type lgssc unregistered [ 3973.143376] LNet: 193331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3973.146076] LNetError: 193331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3973.156325] LNet: Removed LNI 192.168.203.117@tcp [ 3973.485055] Key type .llcrypt unregistered [ 3973.486077] Key type ._llcrypt unregistered [ 3982.397517] Key type ._llcrypt registered [ 3982.398563] Key type .llcrypt registered [ 3982.442146] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3982.928679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3982.944611] alg: No test for adler32 (adler32-zlib) [ 3983.820377] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 3983.918866] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 3985.504163] Key type lgssc registered [ 3985.933046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3988.724296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3988.732992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3989.902520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3991.153901] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3993.791037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3993.915499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3995.122720] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3996.722202] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3997.773172] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3999.914671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4000.034607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4001.062480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4001.776164] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4003.427211] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4009.441996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.444260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4009.447107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4009.451472] Lustre: Skipped 1 previous similar message [ 4013.364111] LustreError: 197097:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4013.419699] Lustre: server umount lustre-OST0000 complete [ 4019.681765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4019.681992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.682590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4019.688253] Lustre: Skipped 2 previous similar messages [ 4020.803081] LustreError: 197299:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4020.806020] LustreError: 197299:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4020.838434] Lustre: server umount lustre-MDT0000 complete [ 4022.181382] LustreError: 196323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082896 with bad export cookie 16511076517645503761 [ 4022.184322] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4022.188030] LustreError: 196323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4022.261442] LustreError: 197503:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4022.264620] LustreError: 197503:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4022.316214] Lustre: server umount lustre-MDT0001 complete [ 4025.948205] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 4027.044343] Key type lgssc unregistered [ 4027.175406] LNet: 198034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4027.178343] LNetError: 198034:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4027.186297] LNet: Removed LNI 192.168.203.117@tcp [ 4027.530961] Key type .llcrypt unregistered [ 4027.532222] Key type ._llcrypt unregistered [ 4035.234184] Key type ._llcrypt registered [ 4035.235481] Key type .llcrypt registered [ 4035.279563] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 4035.718885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4035.734921] alg: No test for adler32 (adler32-zlib) [ 4036.605442] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 4036.708734] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 4038.296171] Key type lgssc registered [ 4038.711631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4041.425166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4041.433138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4042.595986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.948876] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4046.892626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4047.027660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4048.324154] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4050.181269] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4051.429934] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4053.827706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4053.954326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4055.879854] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4057.806634] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4067.304174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4072.418311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4072.419145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.420985] LustreError: Skipped 1 previous similar message [ 4072.421355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4072.424677] Lustre: Skipped 1 previous similar message [ 4073.966722] LustreError: 201675:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4074.008905] Lustre: server umount lustre-OST0000 complete [ 4077.377351] LustreError: 201876:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4077.379454] LustreError: 201876:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4077.409818] Lustre: server umount lustre-MDT0000 complete [ 4077.538327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4077.539199] LustreError: 201419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4077.542818] Lustre: Skipped 1 previous similar message [ 4077.547986] LustreError: 201419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4078.713538] LustreError: 200879:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082952 with bad export cookie 16562530736005551772 [ 4078.717620] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4078.718899] LustreError: 200879:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4078.793967] LustreError: 202078:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4078.796209] LustreError: 202078:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4078.846924] Lustre: server umount lustre-MDT0001 complete [ 4082.373360] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 4083.360911] Key type lgssc unregistered [ 4083.488441] LNet: 202608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4083.491408] LNetError: 202608:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4083.498317] LNet: Removed LNI 192.168.203.117@tcp [ 4083.817782] Key type .llcrypt unregistered [ 4083.818957] Key type ._llcrypt unregistered [ 4090.699417] Key type ._llcrypt registered [ 4090.700371] Key type .llcrypt registered [ 4090.738419] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 4091.076325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4091.130077] alg: No test for adler32 (adler32-zlib) [ 4091.978717] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 4092.054718] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 4093.640123] Key type lgssc registered [ 4094.006369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4096.653604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4096.659847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4097.820855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4099.485250] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4102.287404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4102.399541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4103.680363] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4105.439851] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4106.645335] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4108.804260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4108.908876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4110.635294] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4113.062574] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4122.215593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4126.264824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4126.290439] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4126.376206] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4128.446377] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4130.422413] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4131.046926] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4142.074228] Lustre: Permanently deactivating lustre-OST0001 [ 4142.085466] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4148.246544] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4151.294831] Lustre: Permanently reactivating lustre-OST0001 [ 4151.302190] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4151.304399] Lustre: Skipped 2 previous similar messages [ 4157.945658] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4157.951261] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4157.954553] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4157.958659] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4157.975766] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4159.972513] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4159.973625] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4159.977041] Lustre: Skipped 2 previous similar messages [ 4165.092381] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4165.094357] Lustre: Skipped 2 previous similar messages [ 4165.548785] LustreError: 207300:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4165.566021] Lustre: server umount lustre-OST0001 complete [ 4168.673194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4168.673301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4168.675923] LustreError: Skipped 1 previous similar message [ 4168.680275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4168.683239] Lustre: Skipped 1 previous similar message [ 4170.211305] LustreError: 205432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.216593] LustreError: 205432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4172.973137] LustreError: 207502:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4172.975547] LustreError: 207502:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4173.010846] Lustre: server umount lustre-OST0000 complete [ 4174.304774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4174.307300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4174.310508] Lustre: Skipped 1 previous similar message [ 4174.312632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4174.314615] Lustre: Skipped 2 previous similar messages [ 4180.352269] LustreError: 207703:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4180.354279] LustreError: 207703:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4180.377765] Lustre: server umount lustre-MDT0000 complete [ 4180.448942] LustreError: 203831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4180.454365] LustreError: 203831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4181.590193] LustreError: 207214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083055 with bad export cookie 8273008273263275017 [ 4181.594392] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4181.594557] LustreError: 207214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4181.709670] Lustre: server umount lustre-MDT0001 complete [ 4185.105146] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 4186.107770] Key type lgssc unregistered [ 4186.230409] LNet: 208436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4186.232939] LNetError: 208436:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4186.241304] LNet: Removed LNI 192.168.203.117@tcp [ 4186.537445] Key type .llcrypt unregistered [ 4186.538469] Key type ._llcrypt unregistered [ 4197.453576] Key type ._llcrypt registered [ 4197.454962] Key type .llcrypt registered [ 4197.493962] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 4203.637268] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 4203.976302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4204.026437] alg: No test for adler32 (adler32-zlib) [ 4204.876367] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 4204.968988] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 4206.552147] Key type lgssc registered [ 4206.929091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4209.709994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4211.870025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4213.947196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4216.002080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4218.209316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4218.232854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4218.238246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.322393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4219.332577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4219.367441] Lustre: lustre-MDT0000: new disk, initializing [ 4219.388808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4219.394333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4220.586342] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4224.576538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4224.596354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4224.629236] Lustre: 212352:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4224.650702] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4224.653095] Lustre: Skipped 1 previous similar message [ 4224.688670] Lustre: lustre-MDT0001: new disk, initializing [ 4224.717540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4224.728026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4224.730986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4225.977983] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4229.064300] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4230.207901] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4232.434587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4232.458197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4232.552052] Lustre: lustre-OST0000: new disk, initializing [ 4232.554310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4232.580420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4234.392374] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4237.475281] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4237.809821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4237.812763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4237.822452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 4238.799360] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4238.865522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4240.118856] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4240.187854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4242.912941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4242.915758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4242.920868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4244.962854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4244.963452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4244.967463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4246.893082] LustreError: 215294:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 4246.930539] Lustre: server umount lustre-OST0000 complete [ 4250.081545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4250.082193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4250.085814] Lustre: Skipped 1 previous similar message [ 4250.089473] Lustre: Skipped 2 previous similar messages [ 4254.274855] LustreError: 215496:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4254.277263] LustreError: 215496:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4254.308413] Lustre: server umount lustre-MDT0000 complete [ 4255.201065] LustreError: 213323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4255.208627] LustreError: 213323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4255.586230] LustreError: 212388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083129 with bad export cookie 8337327389147044737 [ 4255.589363] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4255.590080] LustreError: 212388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4255.655378] LustreError: 215697:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4255.657928] LustreError: 215697:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4255.708923] Lustre: server umount lustre-MDT0001 complete [ 4258.534658] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 06:58:52 (1769083132) [ 4260.715994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4260.848803] LustreError: 216367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4260.871729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4262.177863] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4264.942813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4265.050496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4266.318667] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4268.118049] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4269.319379] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4271.567338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4273.625593] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4275.379376] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4286.659043] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4294.200648] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4301.755051] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4311.339739] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4323.338103] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4323.340256] Lustre: Skipped 1 previous similar message [ 4340.502856] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4340.505724] Lustre: Skipped 1 previous similar message [ 4374.915978] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4374.925175] Lustre: Skipped 4 previous similar messages [ 4443.892631] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 4443.897216] Lustre: Skipped 8 previous similar messages [ 4449.763484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.763651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4449.764770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4449.768571] Lustre: Skipped 1 previous similar message [ 4451.828110] LustreError: 219867:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4451.830588] LustreError: 219867:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4451.934532] Lustre: server umount lustre-OST0000 complete [ 4454.369384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4454.372923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.378338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4454.381536] Lustre: Skipped 1 previous similar message [ 4460.240902] LustreError: 220068:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4460.243913] LustreError: 220068:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4460.305060] Lustre: server umount lustre-MDT0000 complete [ 4462.595435] LustreError: 216392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083336 with bad export cookie 8337327389147045612 [ 4462.597759] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4462.608473] LustreError: 216392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4462.836810] Lustre: server umount lustre-MDT0001 complete [ 4469.131653] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 4470.626614] Key type lgssc unregistered [ 4470.784665] LNet: 220802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4470.787461] LNetError: 220802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4470.795580] LNet: Removed LNI 192.168.203.117@tcp [ 4471.322597] Key type .llcrypt unregistered [ 4471.323754] Key type ._llcrypt unregistered [ 4485.784751] Key type ._llcrypt registered [ 4485.786091] Key type .llcrypt registered [ 4485.843283] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 4486.362892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4486.426458] alg: No test for adler32 (adler32-zlib) [ 4487.355259] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 4487.473758] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 4489.096181] Key type lgssc registered [ 4489.831646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4493.894118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4493.906086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4495.193838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4497.327484] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4502.043949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4502.287791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4504.260639] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4507.427880] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4509.459467] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4512.902217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4513.084177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4516.136464] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4518.953914] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4532.707763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4532.708969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4532.712948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4532.722026] Lustre: Skipped 1 previous similar message [ 4533.509279] LustreError: 224573:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4533.589809] Lustre: server umount lustre-OST0000 complete [ 4537.827823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4537.833406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4537.841856] Lustre: Skipped 2 previous similar messages [ 4537.854739] Lustre: Skipped 1 previous similar message [ 4542.946298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4542.949518] Lustre: Skipped 1 previous similar message [ 4548.070215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4548.074631] Lustre: Skipped 1 previous similar message [ 4550.112174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4550.163738] LustreError: 224779:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4550.167215] LustreError: 224779:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4550.217370] Lustre: server umount lustre-MDT0000 complete [ 4552.441779] LustreError: 224424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083426 with bad export cookie 1617318513732000254 [ 4552.446957] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4552.450458] LustreError: 224424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4552.538318] LustreError: 224981:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4552.545753] LustreError: 224981:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4552.720665] Lustre: server umount lustre-MDT0001 complete [ 4559.093695] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 4560.357973] Key type lgssc unregistered [ 4560.497462] LNet: 225511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4560.502859] LNetError: 225511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4560.516412] LNet: Removed LNI 192.168.203.117@tcp [ 4560.862593] Key type .llcrypt unregistered [ 4560.863659] Key type ._llcrypt unregistered [ 4571.980646] Key type ._llcrypt registered [ 4571.985320] Key type .llcrypt registered [ 4572.037444] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 4572.623930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4572.636673] alg: No test for adler32 (adler32-zlib) [ 4573.533403] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 4573.634353] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 4575.232245] Key type lgssc registered [ 4575.643527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4841.766227] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 4843.910423] Key type lgssc unregistered [ 4844.106668] LNet: 227473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4844.110754] LNetError: 227473:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4844.129294] LNet: Removed LNI 192.168.203.117@tcp [ 4844.817507] Key type .llcrypt unregistered [ 4844.819796] Key type ._llcrypt unregistered [ 4863.195891] Key type ._llcrypt registered [ 4863.198944] Key type .llcrypt registered [ 4863.286851] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 4876.085277] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 4876.937848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4876.951875] alg: No test for adler32 (adler32-zlib) [ 4877.865473] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 4877.973309] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 4879.592186] Key type lgssc registered [ 4880.491396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4886.228644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4890.308533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4894.274063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4898.520803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4899.821115] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4627 sec ========== 07:09:32 (1769083772) [ 4901.155749] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:09:34 (1769083774) === [ 4903.417717] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:09:36 (1769083776) === [ 4924.520724] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 4925.800342] Key type lgssc unregistered [ 4925.950860] LNet: 232659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4925.956159] LNetError: 232659:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4925.967677] LNet: Removed LNI 192.168.203.117@tcp [ 4926.361503] Key type .llcrypt unregistered [ 4926.364419] Key type ._llcrypt unregistered