[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 458490829 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003178] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009007] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011047] Yama: becoming mindful. [ 0.012027] SELinux: Initializing. [ 0.013075] *** VALIDATE selinux *** [ 0.022558] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027184] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028175] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030128] *** VALIDATE tmpfs *** [ 0.031499] *** VALIDATE proc *** [ 0.033150] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036185] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039021] Spectre V2 : User space: Vulnerable [ 0.040004] Speculative Store Bypass: Vulnerable [ 0.043168] debug: unmapping init [mem 0xffffffff98259000-0xffffffff98260fff] [ 0.045724] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046488] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047010] ... version: 2 [ 0.047816] ... bit width: 48 [ 0.048006] ... generic registers: 4 [ 0.048763] ... value mask: 0000ffffffffffff [ 0.049008] ... max period: 00007fffffffffff [ 0.050007] ... fixed-purpose events: 3 [ 0.050759] ... event mask: 000000070000000f [ 0.051270] rcu: Hierarchical SRCU implementation. [ 0.053539] smp: Bringing up secondary CPUs ... [ 0.054618] x86: Booting SMP configuration: [ 0.055015] .... node #0, CPUs: #1 #2 #3 [ 0.058015] smp: Brought up 1 node, 4 CPUs [ 0.059891] smpboot: Max logical packages: 1 [ 0.060007] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147016] node 0 deferred pages initialised in 86ms [ 0.149531] devtmpfs: initialized [ 0.150235] x86/mm: Memory block size: 128MB [ 0.152924] gcov: version magic: 0x41383552 [ 0.153581] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154063] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155175] pinctrl core: initialized pinctrl subsystem [ 0.156109] [ 0.156558] ************************************************************* [ 0.157009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158006] ** ** [ 0.159006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160006] ** ** [ 0.161006] ** This means that this kernel is built to expose internal ** [ 0.162007] ** IOMMU data structures, which may compromise security on ** [ 0.163006] ** your system. ** [ 0.164006] ** ** [ 0.165008] ** If you see this message and you are not debugging the ** [ 0.166009] ** kernel, report this immediately to your vendor! ** [ 0.167006] ** ** [ 0.168008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169006] ************************************************************* [ 0.170485] NET: Registered protocol family 16 [ 0.171424] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173043] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174334] cpuidle: using governor menu [ 0.175000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178304] PCI: Using configuration type 1 for base access [ 0.180121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.187160] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.189029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192104] cryptd: max_cpu_qlen set to 1000 [ 0.194063] ACPI: Added _OSI(Module Device) [ 0.196012] ACPI: Added _OSI(Processor Device) [ 0.197011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.203831] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209505] ACPI: Interpreter enabled [ 0.211047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.212009] ACPI: Using IOAPIC for interrupt routing [ 0.215097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.218405] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.225395] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.227017] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.229008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.231040] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.233954] acpiphp: Slot [2] registered [ 0.236084] acpiphp: Slot [3] registered [ 0.237048] acpiphp: Slot [4] registered [ 0.237872] acpiphp: Slot [5] registered [ 0.239065] acpiphp: Slot [6] registered [ 0.240094] acpiphp: Slot [7] registered [ 0.242086] acpiphp: Slot [8] registered [ 0.243057] acpiphp: Slot [9] registered [ 0.245245] acpiphp: Slot [10] registered [ 0.246087] acpiphp: Slot [11] registered [ 0.248104] acpiphp: Slot [12] registered [ 0.249060] acpiphp: Slot [13] registered [ 0.250067] acpiphp: Slot [14] registered [ 0.252066] acpiphp: Slot [15] registered [ 0.253061] acpiphp: Slot [16] registered [ 0.255089] acpiphp: Slot [17] registered [ 0.256061] acpiphp: Slot [18] registered [ 0.257070] acpiphp: Slot [19] registered [ 0.259060] acpiphp: Slot [20] registered [ 0.260075] acpiphp: Slot [21] registered [ 0.261061] acpiphp: Slot [22] registered [ 0.263082] acpiphp: Slot [23] registered [ 0.264057] acpiphp: Slot [24] registered [ 0.265074] acpiphp: Slot [25] registered [ 0.267057] acpiphp: Slot [26] registered [ 0.268062] acpiphp: Slot [27] registered [ 0.269069] acpiphp: Slot [28] registered [ 0.271066] acpiphp: Slot [29] registered [ 0.272095] acpiphp: Slot [30] registered [ 0.274076] acpiphp: Slot [31] registered [ 0.275057] PCI host bridge to bus 0000:00 [ 0.277015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.279013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.281015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.283015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.286015] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.288016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.290165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.294401] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.298599] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.306662] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.312077] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.315017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.318024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.320012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.321488] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.323621] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.326041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.328444] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.332010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.348968] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.352020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.357816] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.365012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.373015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.389016] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.398481] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.404022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.412032] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.429018] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.442382] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.449018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.454019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.470019] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.481874] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.489037] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.498023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.523020] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.538103] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.544020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.554027] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.571022] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.583752] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.589030] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.594015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.611038] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.622176] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.625688] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.628344] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.629288] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.631176] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.634150] iommu: Default domain type: Passthrough [ 0.636391] SCSI subsystem initialized [ 0.637121] ACPI: bus type USB registered [ 0.638074] usbcore: registered new interface driver usbfs [ 0.640053] usbcore: registered new interface driver hub [ 0.641061] usbcore: registered new device driver usb [ 0.643382] pps_core: LinuxPPS API ver. 1 registered [ 0.644017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.648066] PTP clock support registered [ 0.649359] EDAC MC: Ver: 3.0.0 [ 0.650112] PCI: Using ACPI for IRQ routing [ 0.652160] NetLabel: Initializing [ 0.653007] NetLabel: domain hash size = 128 [ 0.654010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.656078] NetLabel: unlabeled traffic allowed by default [ 0.659099] vgaarb: loaded [ 0.661255] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.663008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.669005] clocksource: Switched to clocksource kvm-clock [ 0.771720] VFS: Disk quotas dquot_6.6.0 [ 0.773857] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.776171] *** VALIDATE ramfs *** [ 0.777246] *** VALIDATE hugetlbfs *** [ 0.778658] pnp: PnP ACPI init [ 0.780742] pnp: PnP ACPI: found 6 devices [ 0.798836] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.800780] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.802489] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.803665] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.805598] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.807102] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.809111] NET: Registered protocol family 2 [ 0.810856] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.814605] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.817048] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.820698] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.823906] TCP: Hash tables configured (established 65536 bind 65536) [ 0.826562] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.829938] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.832496] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.835181] NET: Registered protocol family 1 [ 0.837655] RPC: Registered named UNIX socket transport module. [ 0.839631] RPC: Registered udp transport module. [ 0.841054] RPC: Registered tcp transport module. [ 0.842534] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.844146] NET: Registered protocol family 44 [ 0.844929] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.846563] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.848326] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.850172] PCI: CLS 0 bytes, default 64 [ 0.851786] Unpacking initramfs... [ 2.235424] debug: unmapping init [mem 0xffff89ea7cc54000-0xffff89ea7ffbffff] [ 2.240784] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.242487] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.245599] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.713908] Initialise system trusted keyrings [ 2.715432] Key type blacklist registered [ 2.720252] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.729643] zbud: loaded [ 2.732647] *** VALIDATE nfs *** [ 2.733779] *** VALIDATE nfs4 *** [ 2.735274] pstore: using deflate compression [ 2.738589] Platform Keyring initialized [ 2.895456] NET: Registered protocol family 38 [ 2.907973] Key type asymmetric registered [ 2.909867] Asymmetric key parser 'x509' registered [ 2.912103] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.915606] io scheduler mq-deadline registered [ 2.917743] io scheduler kyber registered [ 2.919892] io scheduler bfq registered [ 2.924234] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.927779] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.937401] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.945931] ACPI: Power Button [PWRF] [ 3.166217] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.298538] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.504210] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.651493] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.969890] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.005936] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.038685] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.042625] Non-volatile memory driver v1.3 [ 4.044096] Linux agpgart interface v0.103 [ 4.074097] virtio_blk virtio1: [vda] 134040 512-byte logical blocks (68.6 MB/65.4 MiB) [ 4.077996] vda: detected capacity change from 0 to 68628480 [ 4.101876] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.104663] vdb: detected capacity change from 0 to 1073741824 [ 4.126836] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.129019] vdc: detected capacity change from 0 to 2621440000 [ 4.146382] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.149503] vdd: detected capacity change from 0 to 2621440000 [ 4.164715] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.167499] vde: detected capacity change from 0 to 4294967296 [ 4.184624] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.187642] vdf: detected capacity change from 0 to 4294967296 [ 4.198992] libphy: Fixed MDIO Bus: probed [ 4.207803] usbcore: registered new interface driver usbserial_generic [ 4.210459] usbserial: USB Serial support registered for generic [ 4.212318] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.216408] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.218405] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.221166] mousedev: PS/2 mouse device common for all mice [ 4.224238] rtc_cmos 00:05: RTC can wake from S4 [ 4.230134] rtc_cmos 00:05: registered as rtc0 [ 4.233325] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.237262] intel_pstate: CPU model not supported [ 4.237440] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.243550] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.249925] hid: raw HID events driver (C) Jiri Kosina [ 4.255974] usbcore: registered new interface driver usbhid [ 4.257630] usbhid: USB HID core driver [ 4.257787] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.258660] drop_monitor: Initializing network drop monitor service [ 4.263035] Initializing XFRM netlink socket [ 4.264869] NET: Registered protocol family 10 [ 4.267607] Segment Routing with IPv6 [ 4.269098] NET: Registered protocol family 17 [ 4.270868] mpls_gso: MPLS GSO support [ 4.276876] RAS: Correctable Errors collector initialized. [ 4.279075] AVX version of gcm_enc/dec engaged. [ 4.280505] AES CTR mode by8 optimization enabled [ 4.385085] sched_clock: Marking stable (4385035250, 0)->(5167196184, -782160934) [ 4.391396] registered taskstats version 1 [ 4.394917] Loading compiled-in X.509 certificates [ 4.399255] zswap: loaded using pool lzo/zbud [ 4.438885] Key type big_key registered [ 4.460318] Key type encrypted registered [ 4.461822] ima: No TPM chip found, activating TPM-bypass! [ 4.464510] ima: Allocated hash algorithm: sha1 [ 4.467243] ima: No architecture policies found [ 4.469087] evm: Initialising EVM extended attributes: [ 4.471745] evm: security.selinux [ 4.474624] evm: security.ima [ 4.477154] evm: security.capability [ 4.478148] evm: HMAC attrs: 0x1 [ 4.480953] rtc_cmos 00:05: setting system clock to 2025-12-29 18:54:53 UTC (1767034493) [ 4.488168] debug: unmapping init [mem 0xffffffff99203000-0xffffffff993fffff] [ 4.495465] debug: unmapping init [mem 0xffffffff97f82000-0xffffffff98258fff] [ 4.499396] Write protecting the kernel read-only data: 28672k [ 4.502635] debug: unmapping init [mem 0xffffffff96603000-0xffffffff967fffff] [ 4.508272] debug: unmapping init [mem 0xffffffff96f14000-0xffffffff96ffffff] [ 4.539475] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.546790] systemd[1]: Detected virtualization kvm. [ 4.548173] systemd[1]: Detected architecture x86-64. [ 4.549448] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.580478] systemd[1]: No hostname configured. [ 4.582017] systemd[1]: Set hostname to . [ 4.584219] random: systemd: uninitialized urandom read (16 bytes read) [ 4.586320] systemd[1]: Initializing machine ID from random generator. [ 4.790960] random: systemd: uninitialized urandom read (16 bytes read) [ 4.794412] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.801818] random: systemd: uninitialized urandom read (16 bytes read) [ 4.804988] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.811946] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.767988] device-mapper: uevent: version 1.0.3 [ 5.769686] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.738892] virtio_net virtio0 ens2: renamed from eth0 [ 6.924702] random: fast init done [ 7.057762] scsi host0: ata_piix [ 7.157171] scsi host1: ata_piix [ 7.159750] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.169620] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.997525] random: crng init done [ 12.001278] random: 7 urandom warning(s) missed due to ratelimiting [ 12.889834] dracut-initqueue[574]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.498458] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.655291] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.091220] SELinux: Disabled at runtime. [ 17.160464] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.190152] systemd[1]: Detected virtualization kvm. [ 17.192632] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.100322] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.102834] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.112920] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.116789] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.120277] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.130278] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.153026] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 18.578106] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 19.416669] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.043254] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.159982] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.833679] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.114098] EDAC sbridge: Ver: 1.1.2 [ 22.189437] hrtimer: interrupt took 6426386 ns [ 24.608263] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 25.258300] NFS: Registering the id_resolver key type [ 25.260276] Key type id_resolver registered [ 25.262425] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg326-server login: [ 55.754717] libcfs: loading out-of-tree module taints kernel. [ 55.771295] Key type ._llcrypt registered [ 55.772771] Key type .llcrypt registered [ 55.810322] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_hostid [ 62.579116] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 63.123032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 63.130500] alg: No test for adler32 (adler32-zlib) [ 64.123951] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 64.429342] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 66.040227] Key type lgssc registered [ 66.614157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.622703] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 72.983545] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.829328] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 78.307456] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.887779] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.316415] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 91.116542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.145088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 91.154326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 92.248705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 92.262749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 92.299065] Lustre: lustre-MDT0000: new disk, initializing [ 92.326891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 92.334244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 93.718373] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 98.971577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.995489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 99.022792] Lustre: 6470:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 99.034484] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 99.037222] Lustre: Skipped 1 previous similar message [ 99.075866] Lustre: lustre-MDT0001: new disk, initializing [ 99.096807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 99.104940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 99.107309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 100.379827] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 102.718906] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 106.035744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.059209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 106.131960] Lustre: lustre-OST0000: new disk, initializing [ 106.133688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 106.148259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 108.062854] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.596444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 111.600986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 111.618296] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 113.315178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.338076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 113.364162] Lustre: lustre-OST0001: new disk, initializing [ 113.365466] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 113.377508] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 115.177082] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 120.832829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 120.836184] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 120.861867] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 121.197163] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 128.729839] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 138.734390] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing check_logdir /tmp/testlogs/ [ 142.087485] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing yml_node [ 146.111595] Lustre: DEBUG MARKER: Client: 2.17.0.RC3 [ 148.457956] Lustre: DEBUG MARKER: MDS: 2.17.0.RC3 [ 150.384381] Lustre: DEBUG MARKER: OSS: 2.17.0.RC3 [ 151.359568] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Dec 29 13:57:20 EST 2025 [ 158.904626] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 159.546489] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 172.002337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 172.002617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 172.007250] Lustre: Skipped 1 previous similar message [ 172.012926] Lustre: Skipped 3 previous similar messages [ 177.121055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 177.123995] Lustre: Skipped 3 previous similar messages [ 182.240851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 182.242996] Lustre: Skipped 3 previous similar messages [ 186.336196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 186.380644] LustreError: 12170:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 186.412315] Lustre: server umount lustre-MDT0000 complete [ 187.928735] LustreError: 6462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767034676 with bad export cookie 4991981144172103055 [ 187.930351] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 187.933241] LustreError: 6462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 192.480684] LustreError: 9119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 192.480796] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 192.484459] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 192.486937] LustreError: 9119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 192.491971] Lustre: Skipped 3 previous similar messages [ 197.601786] LustreError: 9119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 197.602724] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 197.608665] LustreError: 9119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 197.610988] Lustre: Skipped 2 previous similar messages [ 202.208161] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 202.273712] LustreError: 12374:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 202.276415] LustreError: 12374:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 202.331106] Lustre: server umount lustre-MDT0001 complete [ 204.308771] LustreError: 12576:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 204.311856] LustreError: 12576:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 204.347980] Lustre: server umount lustre-OST0000 complete [ 206.387970] LustreError: 12776:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 206.390880] LustreError: 12776:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 206.450255] Lustre: server umount lustre-OST0001 complete [ 209.653219] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_hostid [ 214.124851] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 218.411954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 221.445938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 224.416202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 227.511151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 230.683809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 230.709578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.822862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 230.842854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 230.885144] Lustre: lustre-MDT0000: new disk, initializing [ 230.921986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.930860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 232.629928] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 237.962826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 237.991840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 238.041382] Lustre: 14971:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 238.164200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 239.792551] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 243.536665] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 245.239075] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 247.999207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 248.035975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 248.158152] Lustre: lustre-OST0000: new disk, initializing [ 248.160223] Lustre: Skipped 1 previous similar message [ 248.163759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 248.166474] Lustre: Skipped 2 previous similar messages [ 248.208304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 248.211229] Lustre: Skipped 1 previous similar message [ 249.615964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 249.620665] Lustre: Skipped 1 previous similar message [ 249.623077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 249.668342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 250.743100] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 254.381377] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 256.035283] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 256.135901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 257.836640] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 257.928437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 260.068474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 260.070885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 260.072503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 260.074633] Lustre: Skipped 1 previous similar message [ 273.376761] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 273.411068] LustreError: 17907:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 273.415781] LustreError: 17907:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 273.477432] Lustre: server umount lustre-OST0000 complete [ 275.425517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 275.431075] Lustre: Skipped 1 previous similar message [ 280.546417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 280.549682] Lustre: Skipped 8 previous similar messages [ 281.556289] LustreError: 18110:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 281.560717] LustreError: 18110:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 281.601214] Lustre: server umount lustre-MDT0000 complete [ 283.437383] LustreError: 15750:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767034772 with bad export cookie 4991981144172104588 [ 283.441431] LustreError: 15750:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 283.441706] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 283.602414] Lustre: server umount lustre-MDT0001 complete [ 285.281902] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 13:59:34 (1767034774) [ 288.223814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 288.469563] LustreError: 18837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 288.506750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 290.358176] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 293.857150] LustreError: 18837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.123292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 296.114900] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 298.840227] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 300.747909] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 304.386471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 307.199315] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 310.197750] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 322.017953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 322.021494] LustreError: Skipped 1 previous similar message [ 322.023420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.031580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 324.066744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 325.427366] LustreError: 21080:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 325.431068] LustreError: 21080:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 325.506791] Lustre: server umount lustre-OST0000 complete [ 333.740682] Lustre: server umount lustre-MDT0000 complete [ 334.315760] LustreError: 19548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 334.327116] LustreError: 19548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 336.084915] LustreError: 18817:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767034825 with bad export cookie 4991981144172105470 [ 336.088271] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 336.092468] LustreError: 18817:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 336.297354] Lustre: server umount lustre-MDT0001 complete [ 342.346524] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 344.092296] Key type lgssc unregistered [ 344.285057] LNet: 22018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.289726] LNetError: 22018:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 344.302516] LNet: Removed LNI 192.168.203.126@tcp [ 344.860339] Key type .llcrypt unregistered [ 344.861799] Key type ._llcrypt unregistered [ 358.194906] Key type ._llcrypt registered [ 358.198278] Key type .llcrypt registered [ 358.259670] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 358.722216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 358.773503] alg: No test for adler32 (adler32-zlib) [ 359.753816] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 359.937514] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 361.552167] Key type lgssc registered [ 362.328674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 366.727084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 366.738668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 368.059409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 370.262923] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 375.386576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 375.585790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 377.727361] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 380.703424] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 382.678810] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 386.031950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 386.275724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.362649] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 392.340207] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 393.448722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 405.985775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 405.986403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 405.989486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 405.993649] Lustre: Skipped 1 previous similar message [ 405.997652] Lustre: Skipped 1 previous similar message [ 407.672610] LustreError: 26090:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 407.729333] Lustre: server umount lustre-OST0000 complete [ 410.033896] LustreError: 26291:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 410.038064] LustreError: 26291:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 410.090591] Lustre: server umount lustre-MDT0000 complete [ 411.107550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 411.108411] LustreError: 25018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.112590] Lustre: Skipped 1 previous similar message [ 411.123780] LustreError: 25018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 412.150744] LustreError: 23370:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767034901 with bad export cookie 6348923864246364321 [ 412.152600] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 412.158121] LustreError: 23370:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 412.249644] LustreError: 26491:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 412.252604] LustreError: 26491:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 412.340156] Lustre: server umount lustre-MDT0001 complete [ 418.011561] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 419.634054] Key type lgssc unregistered [ 419.810873] LNet: 27022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 419.818874] LNetError: 27022:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 419.827419] LNet: Removed LNI 192.168.203.126@tcp [ 420.261751] Key type .llcrypt unregistered [ 420.263379] Key type ._llcrypt unregistered [ 431.992537] Key type ._llcrypt registered [ 431.994343] Key type .llcrypt registered [ 432.052519] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 432.416287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 432.469032] alg: No test for adler32 (adler32-zlib) [ 433.381131] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 433.491249] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 435.088491] Key type lgssc registered [ 435.630165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 439.001438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 439.010247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.243562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.892840] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 445.827525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 446.028798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 447.867817] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 450.600206] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 452.526634] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 459.345412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 459.542886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.398623] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 465.023233] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 465.705363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 476.132812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 476.136105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.145632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 477.154289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 477.154302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 477.162421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.435072] LustreError: 30984:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 480.485429] Lustre: server umount lustre-OST0000 complete [ 482.273160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.277517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 482.280233] Lustre: Skipped 2 previous similar messages [ 487.395018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 487.397517] Lustre: Skipped 1 previous similar message [ 488.269325] LustreError: 31185:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 488.272747] LustreError: 31185:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 488.322834] Lustre: server umount lustre-MDT0000 complete [ 490.133345] LustreError: 30897:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767034979 with bad export cookie 6783464841477288185 [ 490.136761] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 490.140571] LustreError: 30897:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 490.220547] LustreError: 31386:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 490.223230] LustreError: 31386:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 490.291073] Lustre: server umount lustre-MDT0001 complete [ 494.762238] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 496.041218] Key type lgssc unregistered [ 496.201641] LNet: 31917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 496.204604] LNetError: 31917:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 496.211430] LNet: Removed LNI 192.168.203.126@tcp [ 496.567573] Key type .llcrypt unregistered [ 496.569220] Key type ._llcrypt unregistered [ 507.254887] Key type ._llcrypt registered [ 507.256445] Key type .llcrypt registered [ 507.316205] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 507.622312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 507.645321] alg: No test for adler32 (adler32-zlib) [ 508.517762] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 508.625315] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 510.232313] Key type lgssc registered [ 510.694490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 514.175854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 514.184877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.412195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.034688] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 520.513588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 520.690704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 522.376483] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 524.710135] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 526.163471] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 528.925607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 529.066231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.249436] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 533.522104] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 540.328094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 550.882891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 550.884473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.885292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 550.892943] Lustre: Skipped 1 previous similar message [ 553.134624] LustreError: 35555:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 553.174229] Lustre: server umount lustre-OST0000 complete [ 556.002633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.003457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 556.009106] Lustre: Skipped 1 previous similar message [ 556.011199] Lustre: Skipped 1 previous similar message [ 559.009955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 559.012537] Lustre: Skipped 2 previous similar messages [ 560.771114] LustreError: 35756:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 560.773507] LustreError: 35756:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 560.808043] Lustre: server umount lustre-MDT0000 complete [ 562.362705] LustreError: 35757:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035051 with bad export cookie 1807885016632232290 [ 562.366986] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 562.370179] LustreError: 35757:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 562.442915] LustreError: 35959:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 562.445982] LustreError: 35959:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 562.513821] Lustre: server umount lustre-MDT0001 complete [ 566.855296] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 568.070230] Key type lgssc unregistered [ 568.204489] LNet: 36490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 568.207693] LNetError: 36490:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 568.215685] LNet: Removed LNI 192.168.203.126@tcp [ 568.571904] Key type .llcrypt unregistered [ 568.573780] Key type ._llcrypt unregistered [ 579.031264] Key type ._llcrypt registered [ 579.033016] Key type .llcrypt registered [ 579.079713] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 579.457177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 579.505398] alg: No test for adler32 (adler32-zlib) [ 580.404222] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 580.505884] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 582.120446] Key type lgssc registered [ 582.585039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 585.770330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 585.779332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.014935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 588.602354] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 591.993198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 592.158601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 593.833578] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 596.190339] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 597.864224] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 600.517126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 600.669311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 602.804921] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 604.997657] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 612.969332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 618.469270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 618.469278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 618.476364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 618.478850] Lustre: Skipped 1 previous similar message [ 620.777458] Lustre: lustre-OST0000: Not available for connect from 192.168.203.26@tcp (stopping) [ 623.585904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 623.588465] Lustre: Skipped 1 previous similar message [ 625.888472] Lustre: lustre-OST0000: Not available for connect from 192.168.203.26@tcp (stopping) [ 631.010370] Lustre: lustre-OST0000: Not available for connect from 192.168.203.26@tcp (stopping) [ 631.014316] Lustre: Skipped 2 previous similar messages [ 632.288238] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 632.312314] LustreError: 40124:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 632.357271] Lustre: server umount lustre-OST0000 complete [ 645.998461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 645.999400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 646.003341] Lustre: Skipped 2 previous similar messages [ 646.007919] Lustre: Skipped 1 previous similar message [ 652.169021] LustreError: 40504:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 652.171879] LustreError: 40504:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 652.226784] Lustre: server umount lustre-MDT0000 complete [ 653.956790] LustreError: 37840:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035142 with bad export cookie 4444961799746317081 [ 653.960832] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 653.964792] LustreError: 37840:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 660.104442] LustreError: 40706:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 660.107788] LustreError: 40706:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 660.198153] Lustre: server umount lustre-MDT0001 complete [ 664.459859] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 665.629325] Key type lgssc unregistered [ 665.785608] LNet: 41237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.789208] LNetError: 41237:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.801360] LNet: Removed LNI 192.168.203.126@tcp [ 666.157475] Key type .llcrypt unregistered [ 666.159075] Key type ._llcrypt unregistered [ 676.540492] Key type ._llcrypt registered [ 676.542156] Key type .llcrypt registered [ 676.591319] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 676.958292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 677.013372] alg: No test for adler32 (adler32-zlib) [ 677.900370] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 678.003959] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 679.600162] Key type lgssc registered [ 680.042136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 683.179724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 683.189907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 684.396912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 685.959383] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 689.440272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 689.598167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 691.183748] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 693.497123] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 695.081269] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 697.948507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 698.114424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 700.495240] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 702.688402] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 709.415557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 719.592651] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.26@tcp (stopping) [ 719.842786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 719.848776] Lustre: Skipped 2 previous similar messages [ 723.149337] LustreError: 44873:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 723.162287] LustreError: 44873:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000093614b6e) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 724.704164] LustreError: 42605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 724.711927] LustreError: 42605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 725.218281] LustreError: 44873:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 725.220251] LustreError: 44873:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 725.237865] Lustre: server umount lustre-MDT0000 complete [ 726.643412] LustreError: 44877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035215 with bad export cookie 11664320896786331237 [ 726.646670] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 726.647858] LustreError: 44877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 729.824470] LustreError: 42603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 729.830900] LustreError: 42603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 729.831640] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.26@tcp (stopping) [ 729.837255] Lustre: Skipped 3 previous similar messages [ 730.081871] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 731.105082] LustreError: 43816:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 731.105408] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 731.111583] LustreError: 43816:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 731.118301] Lustre: Skipped 1 previous similar message [ 732.804312] LustreError: 45079:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 732.806947] LustreError: 45079:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 732.877447] Lustre: server umount lustre-MDT0001 complete [ 759.264098] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 759.281920] LustreError: 45301:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 759.284339] LustreError: 45301:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 759.334426] Lustre: server umount lustre-OST0000 complete [ 766.232250] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 767.475836] Key type lgssc unregistered [ 767.637409] LNet: 46138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.641044] LNetError: 46138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.650731] LNet: Removed LNI 192.168.203.126@tcp [ 768.068746] Key type .llcrypt unregistered [ 768.070742] Key type ._llcrypt unregistered [ 779.477560] Key type ._llcrypt registered [ 779.478932] Key type .llcrypt registered [ 779.522909] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 779.966492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 779.975632] alg: No test for adler32 (adler32-zlib) [ 780.852190] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 780.949769] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 782.544195] Key type lgssc registered [ 782.992323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 786.173329] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 786.185875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 801.632472] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 806.752234] LustreError: 47483:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.126@tcp: failed processing log, type 4: rc = -110 [ 833.376217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 835.736389] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 837.903624] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1001.049422] LustreError: 48192:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1001.090251] Lustre: server umount lustre-OST0000 complete [ 1006.645690] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 1007.669052] Key type lgssc unregistered [ 1007.798371] LNet: 49030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.800644] LNetError: 49030:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.811569] LNet: Removed LNI 192.168.203.126@tcp [ 1008.110936] Key type .llcrypt unregistered [ 1008.112102] Key type ._llcrypt unregistered [ 1017.604366] Key type ._llcrypt registered [ 1017.605706] Key type .llcrypt registered [ 1017.652924] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 1018.084852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1018.120233] alg: No test for adler32 (adler32-zlib) [ 1019.002993] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1019.103312] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 1020.704158] Key type lgssc registered [ 1021.131900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1024.160923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1024.168866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.387634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1026.870791] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1030.168339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1030.330960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1031.819311] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1033.825838] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1035.199525] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1037.810640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1037.958055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.112785] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1042.143337] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1045.475100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1045.476816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1045.481274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1049.069811] LustreError: 52644:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1049.108131] Lustre: server umount lustre-OST0000 complete [ 1050.593818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1050.594718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1050.598751] Lustre: Skipped 1 previous similar message [ 1055.713835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1055.717753] Lustre: Skipped 2 previous similar messages [ 1056.579438] LustreError: 52845:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1056.581863] LustreError: 52845:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1056.610729] Lustre: server umount lustre-MDT0000 complete [ 1057.974431] LustreError: 52846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035546 with bad export cookie 10984836654586206041 [ 1057.977637] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1057.979050] LustreError: 52846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1058.053523] LustreError: 53047:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1058.056047] LustreError: 53047:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1058.106803] Lustre: server umount lustre-MDT0001 complete [ 1062.022285] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 1063.190492] Key type lgssc unregistered [ 1063.331633] LNet: 53578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.334730] LNetError: 53578:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.342386] LNet: Removed LNI 192.168.203.126@tcp [ 1063.687260] Key type .llcrypt unregistered [ 1063.688648] Key type ._llcrypt unregistered [ 1073.782400] Key type ._llcrypt registered [ 1073.783841] Key type .llcrypt registered [ 1073.826713] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 1074.165283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1074.179549] alg: No test for adler32 (adler32-zlib) [ 1075.052656] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1075.147854] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 1076.744219] Key type lgssc registered [ 1077.193174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1080.249759] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1080.260604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1095.712334] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1100.832192] LustreError: 54921:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.126@tcp: failed processing log, type 4: rc = -110 [ 1127.456240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.514145] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1131.423117] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1133.879753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.082250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1134.110312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1135.539409] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1138.666702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1138.831292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1140.300783] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1142.174956] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1143.498627] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1144.289146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1144.289495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.291957] LustreError: Skipped 1 previous similar message [ 1144.292531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1144.295770] Lustre: Skipped 1 previous similar message [ 1148.386461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1148.388772] Lustre: Skipped 1 previous similar message [ 1150.316532] LustreError: 57192:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1150.355549] Lustre: server umount lustre-OST0000 complete [ 1164.509508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.514350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1164.516421] Lustre: Skipped 2 previous similar messages [ 1169.889052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.891520] Lustre: Skipped 1 previous similar message [ 1170.685362] LustreError: 57575:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1170.687202] LustreError: 57575:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1170.714172] Lustre: server umount lustre-MDT0000 complete [ 1171.923246] LustreError: 54929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035660 with bad export cookie 1085582027550859475 [ 1171.926167] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1171.929259] LustreError: 54929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1186.272134] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1186.304476] LustreError: 57777:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1186.306756] LustreError: 57777:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1186.359669] Lustre: server umount lustre-MDT0001 complete [ 1189.798096] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 1190.808251] Key type lgssc unregistered [ 1190.935366] LNet: 58310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.937574] LNetError: 58310:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.945282] LNet: Removed LNI 192.168.203.126@tcp [ 1191.235267] Key type .llcrypt unregistered [ 1191.236583] Key type ._llcrypt unregistered [ 1200.167742] Key type ._llcrypt registered [ 1200.168800] Key type .llcrypt registered [ 1200.207035] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 1200.631084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1200.639133] alg: No test for adler32 (adler32-zlib) [ 1201.495276] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1201.586983] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 1203.168165] Key type lgssc registered [ 1203.550877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1206.177631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1206.183945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.343608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.694206] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1211.628608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1211.769577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1213.140240] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1215.087500] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1216.327833] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1218.660465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1218.782282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1220.826373] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1222.710792] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1233.127565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1237.473232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1237.475654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.480433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1238.499339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1238.499616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1238.499905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.508220] Lustre: Skipped 1 previous similar message [ 1241.390341] LustreError: 61944:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1241.424276] Lustre: server umount lustre-OST0000 complete [ 1243.617524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.618360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1243.620910] Lustre: Skipped 1 previous similar message [ 1243.624358] Lustre: Skipped 1 previous similar message [ 1248.737452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1248.739964] Lustre: Skipped 1 previous similar message [ 1248.767565] LustreError: 62145:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1248.769404] LustreError: 62145:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1248.799156] Lustre: server umount lustre-MDT0000 complete [ 1250.091310] LustreError: 62148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035739 with bad export cookie 4469034711413568086 [ 1250.095698] LustreError: 62148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1250.095733] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1250.160022] LustreError: 62349:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1250.162325] LustreError: 62349:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1250.213493] Lustre: server umount lustre-MDT0001 complete [ 1253.802527] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 1254.914378] Key type lgssc unregistered [ 1255.043634] LNet: 62879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.046738] LNetError: 62879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1255.056447] LNet: Removed LNI 192.168.203.126@tcp [ 1255.372856] Key type .llcrypt unregistered [ 1255.374499] Key type ._llcrypt unregistered [ 1265.556925] Key type ._llcrypt registered [ 1265.558507] Key type .llcrypt registered [ 1265.599231] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 1266.073284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1266.091146] alg: No test for adler32 (adler32-zlib) [ 1266.976385] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1267.072445] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 1268.665402] Key type lgssc registered [ 1269.110324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1272.530763] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 14:16:01 (1767035761) [ 1275.335065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1275.344609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.597891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.138924] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1281.030115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1281.164193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1282.648251] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1284.664230] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1285.920747] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1288.270177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1288.387217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.158980] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1291.963873] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1301.735751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1304.416788] Lustre: Failing over lustre-MDT0000 [ 1304.603791] LustreError: 66807:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1304.625079] Lustre: server umount lustre-MDT0000 complete [ 1307.107367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.107603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1307.111854] LustreError: 66645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1307.120868] LustreError: 66645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1307.760606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1307.810157] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1307.866155] Lustre: *** cfs_fail_loc=135, val=0*** [ 1307.908439] LustreError: 67319:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1307.911081] LustreError: 67319:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.126@tcp: cfg command failed: rc = -2 [ 1307.914261] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1307.914261] [ 1307.918360] LustreError: MGC192.168.203.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1307.923686] LustreError: 67307:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1307.926941] LustreError: 67307:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1307.930605] LustreError: 67307:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 1307.933397] LustreError: 67307:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 1307.943967] Lustre: server umount lustre-MDT0000 complete [ 1307.945152] LustreError: 67307:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1310.245175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1310.294442] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1310.403352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1310.430427] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1311.818966] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1312.480441] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1315.814097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1315.816209] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1315.835509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1317.344894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1317.347482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.351286] Lustre: Skipped 2 previous similar messages [ 1317.352927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1320.930191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1320.931106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1320.933485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.863229] LustreError: 68128:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1322.865895] LustreError: 68128:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1322.901195] Lustre: server umount lustre-OST0000 complete [ 1326.049879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.050032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1326.050629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.050639] Lustre: Skipped 1 previous similar message [ 1326.053059] Lustre: Skipped 1 previous similar message [ 1330.238737] LustreError: 68329:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1330.241141] LustreError: 68329:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1330.266982] Lustre: server umount lustre-MDT0000 complete [ 1331.168837] LustreError: 64540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1331.173036] LustreError: 64540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1331.541144] LustreError: 64520:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035820 with bad export cookie 12310843220411552661 [ 1331.544452] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1331.544924] LustreError: 64520:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1331.656427] Lustre: server umount lustre-MDT0001 complete [ 1335.056770] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 1336.127682] Key type lgssc unregistered [ 1336.251382] LNet: 69062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.253321] LNetError: 69062:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.261363] LNet: Removed LNI 192.168.203.126@tcp [ 1336.556268] Key type .llcrypt unregistered [ 1336.557541] Key type ._llcrypt unregistered [ 1345.342781] Key type ._llcrypt registered [ 1345.343926] Key type .llcrypt registered [ 1345.386789] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 1345.746371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1345.777117] alg: No test for adler32 (adler32-zlib) [ 1346.640464] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1346.735228] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 1348.328343] Key type lgssc registered [ 1348.780799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1351.584300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1351.592076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.755198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.072449] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1356.945360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1357.075965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1358.399901] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1360.184387] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1361.357253] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1363.652341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1363.768672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.705749] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1367.588962] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1377.063273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1380.183337] Lustre: Failing over lustre-MDT0000 [ 1380.305069] LustreError: 72696:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1380.335541] Lustre: server umount lustre-MDT0000 complete [ 1382.371103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.373147] LustreError: 71793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.374880] Lustre: Skipped 2 previous similar messages [ 1382.380614] LustreError: 71793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1383.141905] LustreError: 70426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.627659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.666613] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1383.724933] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1383.778534] LustreError: 73204:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 1383.780215] LustreError: 73204:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.126@tcp: cfg command failed: rc = -9 [ 1383.782904] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1383.782904] [ 1383.785634] LustreError: MGC192.168.203.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1383.790515] LustreError: 73192:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1383.793523] LustreError: 73192:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 1383.796686] LustreError: 73192:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 1383.798923] LustreError: 73192:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 1383.807438] Lustre: server umount lustre-MDT0000 complete [ 1383.809137] LustreError: 73192:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1386.193463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.239323] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1386.341655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1386.375443] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1387.712438] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1388.256483] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1391.589796] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1391.597538] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1391.615103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1393.632917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1393.636132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.642443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1396.705868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1396.708100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1396.708263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1398.701083] LustreError: 74010:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1398.702918] LustreError: 74010:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1398.741111] Lustre: server umount lustre-OST0000 complete [ 1401.825374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1401.825877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.826757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1401.826763] Lustre: Skipped 1 previous similar message [ 1401.834370] Lustre: Skipped 1 previous similar message [ 1406.151320] LustreError: 74216:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1406.153974] LustreError: 74216:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1406.183430] Lustre: server umount lustre-MDT0000 complete [ 1406.945347] LustreError: 71794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1406.950916] LustreError: 71794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1407.425276] LustreError: 72050:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035896 with bad export cookie 6470410317078289931 [ 1407.427965] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1407.429300] LustreError: 72050:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1407.541694] Lustre: server umount lustre-MDT0001 complete [ 1411.315098] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 1412.432707] Key type lgssc unregistered [ 1412.563383] LNet: 74948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.565370] LNetError: 74948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.574353] LNet: Removed LNI 192.168.203.126@tcp [ 1412.896454] Key type .llcrypt unregistered [ 1412.897407] Key type ._llcrypt unregistered [ 1417.088048] Key type ._llcrypt registered [ 1417.088958] Key type .llcrypt registered [ 1417.132205] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 1417.229399] Key type .llcrypt unregistered [ 1417.231099] Key type ._llcrypt unregistered [ 1426.038064] Key type ._llcrypt registered [ 1426.039508] Key type .llcrypt registered [ 1426.078345] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 1426.388977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1426.431069] alg: No test for adler32 (adler32-zlib) [ 1427.281556] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1427.361969] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 1428.944199] Key type lgssc registered [ 1429.351752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1432.095924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1432.101900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1433.263987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.687474] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1437.824136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1437.968074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1439.433956] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1441.307103] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1442.631085] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1444.998219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1445.136902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.225344] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1449.122022] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1458.471198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1462.752892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1462.755884] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.761256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1463.779291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1463.779397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.785966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1468.272130] LustreError: 79054:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1468.307736] Lustre: server umount lustre-OST0000 complete [ 1474.019164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1474.019182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.019903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1474.019909] Lustre: Skipped 1 previous similar message [ 1474.031656] Lustre: Skipped 1 previous similar message [ 1475.707453] LustreError: 79256:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1475.709224] LustreError: 79256:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1475.735367] Lustre: server umount lustre-MDT0000 complete [ 1477.041302] LustreError: 77504:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035966 with bad export cookie 18377924486610354267 [ 1477.045703] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1477.046456] LustreError: 77504:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1477.116503] LustreError: 79458:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1477.118489] LustreError: 79458:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1477.168850] Lustre: server umount lustre-MDT0001 complete [ 1480.723384] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 1481.821860] Key type lgssc unregistered [ 1481.953471] LNet: 79989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.956433] LNetError: 79989:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.964329] LNet: Removed LNI 192.168.203.126@tcp [ 1482.293968] Key type .llcrypt unregistered [ 1482.295379] Key type ._llcrypt unregistered [ 1490.994433] Key type ._llcrypt registered [ 1490.995389] Key type .llcrypt registered [ 1491.038661] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 1491.551746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1491.574732] alg: No test for adler32 (adler32-zlib) [ 1492.430298] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1492.512084] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 1494.088200] Key type lgssc registered [ 1494.478943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1497.186962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1497.194976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.376385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1499.788851] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1502.802793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1502.942964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1504.276287] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1506.069736] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1507.280576] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1509.595718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1509.713887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1511.550583] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1513.206270] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1523.046621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1528.289876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1528.291455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1528.291830] LustreError: Skipped 1 previous similar message [ 1528.296477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1528.299185] Lustre: Skipped 1 previous similar message [ 1532.205845] LustreError: 83627:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1532.244297] Lustre: server umount lustre-OST0000 complete [ 1533.920991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1533.923676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1533.927741] Lustre: Skipped 1 previous similar message [ 1533.930247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1538.529486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1538.530255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.644429] LustreError: 83828:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1539.646378] LustreError: 83828:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1539.670614] Lustre: server umount lustre-MDT0000 complete [ 1540.921479] LustreError: 82077:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036029 with bad export cookie 12021565921978864915 [ 1540.924927] LustreError: 82077:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1540.925788] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1541.004062] LustreError: 84030:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1541.006219] LustreError: 84030:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1541.068616] Lustre: server umount lustre-MDT0001 complete [ 1544.660115] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 1545.689768] Key type lgssc unregistered [ 1545.821317] LNet: 84561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.824259] LNetError: 84561:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1545.837356] LNet: Removed LNI 192.168.203.126@tcp [ 1546.134715] Key type .llcrypt unregistered [ 1546.136137] Key type ._llcrypt unregistered [ 1554.678351] Key type ._llcrypt registered [ 1554.679291] Key type .llcrypt registered [ 1554.719896] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 1555.147413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1555.161240] alg: No test for adler32 (adler32-zlib) [ 1556.023562] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1556.112613] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 1557.704241] Key type lgssc registered [ 1558.105288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1560.789575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1560.794534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1561.944686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1563.286830] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1566.038991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1566.151036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1567.420383] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1569.094779] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1570.217902] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1572.391202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1572.504763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.400439] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1576.334822] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1586.854204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1592.294259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1592.294836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1592.297368] LustreError: Skipped 1 previous similar message [ 1592.301946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1592.304070] Lustre: Skipped 1 previous similar message [ 1596.272459] LustreError: 88199:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1596.307628] Lustre: server umount lustre-OST0000 complete [ 1602.016920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1602.020081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1602.025202] Lustre: Skipped 1 previous similar message [ 1602.027782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1603.712932] LustreError: 88401:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1603.715239] LustreError: 88401:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1603.740626] Lustre: server umount lustre-MDT0000 complete [ 1605.114355] LustreError: 85912:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036094 with bad export cookie 8030637081397801815 [ 1605.117577] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1605.120244] LustreError: 85912:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1605.188655] LustreError: 88603:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1605.190636] LustreError: 88603:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1605.245163] Lustre: server umount lustre-MDT0001 complete [ 1609.178806] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 1610.290926] Key type lgssc unregistered [ 1610.426569] LNet: 89135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.429586] LNetError: 89135:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.441489] LNet: Removed LNI 192.168.203.126@tcp [ 1610.773516] Key type .llcrypt unregistered [ 1610.774645] Key type ._llcrypt unregistered [ 1619.519499] Key type ._llcrypt registered [ 1619.520686] Key type .llcrypt registered [ 1619.556154] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 1619.932384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1619.955699] alg: No test for adler32 (adler32-zlib) [ 1620.802419] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1620.890903] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 1622.480231] Key type lgssc registered [ 1622.854474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1625.443731] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1625.450046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1640.864410] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1645.984275] LustreError: 90493:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.126@tcp: failed processing log, type 4: rc = -110 [ 1672.608187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1674.705342] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1676.769081] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1678.731952] LustreError: 91350:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1678.764735] Lustre: server umount lustre-OST0000 complete [ 1681.455124] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 14:22:50 (1767036170) [ 1683.665829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.792145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1685.093535] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1687.975899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1688.106891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1689.485956] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1691.420928] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1692.645700] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1694.932250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1695.050189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.018318] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1698.840032] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1709.415696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1718.753339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1718.755497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1718.759437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1719.777754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1719.779266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1719.780411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.629070] LustreError: 94443:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1723.631297] LustreError: 94443:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1723.665377] Lustre: server umount lustre-OST0000 complete [ 1724.897187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.897719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1724.902352] Lustre: Skipped 1 previous similar message [ 1724.904069] Lustre: Skipped 1 previous similar message [ 1730.017098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1730.019285] Lustre: Skipped 2 previous similar messages [ 1735.137031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1735.139028] Lustre: Skipped 1 previous similar message [ 1739.232124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1739.264091] LustreError: 94645:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1739.265751] LustreError: 94645:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1739.291152] Lustre: server umount lustre-MDT0000 complete [ 1740.256865] LustreError: 92018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1740.261553] LustreError: 92018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1740.460289] LustreError: 92000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036229 with bad export cookie 6732940257520487902 [ 1740.462241] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1740.465394] LustreError: 92000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1740.577630] Lustre: server umount lustre-MDT0001 complete [ 1743.866381] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 1744.878636] Key type lgssc unregistered [ 1745.004366] LNet: 95380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.006608] LNetError: 95380:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.013286] LNet: Removed LNI 192.168.203.126@tcp [ 1745.302940] Key type .llcrypt unregistered [ 1745.304395] Key type ._llcrypt unregistered [ 1753.612794] Key type ._llcrypt registered [ 1753.613902] Key type .llcrypt registered [ 1753.647492] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 1754.001741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1754.031736] alg: No test for adler32 (adler32-zlib) [ 1754.875756] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1754.956280] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 1756.544184] Key type lgssc registered [ 1756.898549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1759.370851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1759.376181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.519247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.802750] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1764.642682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1764.767273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1766.086645] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1767.961463] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1769.182254] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1771.418057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1771.534211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1773.398620] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1775.247936] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1785.898051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1795.040936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1795.043722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.048861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1796.066582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1796.067512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.067552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1799.406255] LustreError: 99018:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1799.442595] Lustre: server umount lustre-OST0000 complete [ 1801.185476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.186307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1801.190412] Lustre: Skipped 1 previous similar message [ 1801.193855] Lustre: Skipped 2 previous similar messages [ 1806.305023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1806.307287] Lustre: Skipped 1 previous similar message [ 1811.425065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1811.426877] Lustre: Skipped 1 previous similar message [ 1815.008124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1815.038664] LustreError: 99219:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1815.040489] LustreError: 99219:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1815.064554] Lustre: server umount lustre-MDT0000 complete [ 1816.311488] LustreError: 96729:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036305 with bad export cookie 9292952100918893568 [ 1816.315865] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1816.387747] LustreError: 99421:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1816.389446] LustreError: 99421:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1816.449335] Lustre: server umount lustre-MDT0001 complete [ 1819.919168] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 1820.943086] Key type lgssc unregistered [ 1821.071325] LNet: 99952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.073965] LNetError: 99952:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.083298] LNet: Removed LNI 192.168.203.126@tcp [ 1821.380884] Key type .llcrypt unregistered [ 1821.382478] Key type ._llcrypt unregistered [ 1828.438339] Key type ._llcrypt registered [ 1828.439184] Key type .llcrypt registered [ 1828.472358] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 1828.754446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1828.792318] alg: No test for adler32 (adler32-zlib) [ 1829.647392] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1829.734037] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 1831.312157] Key type lgssc registered [ 1831.658237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1834.059110] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1834.064589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1849.504261] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1854.624167] LustreError: 101152:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.126@tcp: failed processing log, type 4: rc = -110 [ 1881.248151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.099396] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1884.803732] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1887.005865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.118438] LustreError: MGC192.168.203.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1887.124256] LustreError: 102062:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1887.127502] LustreError: 102062:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1887.130794] LustreError: 102062:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 1887.160437] Lustre: server umount lustre-MDT0000 complete [ 1887.161631] LustreError: 102062:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1900.000478] LustreError: 102494:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1900.033314] Lustre: server umount lustre-OST0000 complete [ 1902.941596] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_hostid [ 1905.985983] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 1909.085791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1911.129863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1913.110118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1915.177492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1917.499702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1917.519138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.605939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1917.622610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1917.663568] Lustre: lustre-MDT0000: new disk, initializing [ 1917.691293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1917.699029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1919.151393] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1923.575556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1923.600113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1923.635075] Lustre: 104827:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1923.660239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1923.663113] Lustre: Skipped 1 previous similar message [ 1923.702319] Lustre: lustre-MDT0001: new disk, initializing [ 1923.736117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1923.750032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1923.753782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1925.143598] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1928.373992] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1929.628399] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1932.009277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1932.032278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1932.129649] Lustre: lustre-OST0000: new disk, initializing [ 1932.132336] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1932.166586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1934.222350] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1935.160996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1935.165175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1935.198344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1937.432707] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1938.914100] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1938.987945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1940.367360] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1940.450089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1943.152145] LustreError: 107763:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1943.154707] LustreError: 107763:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1943.195073] Lustre: server umount lustre-OST0000 complete [ 1944.544757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1944.547242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1944.551809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.569152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.569729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1950.689500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1950.691870] Lustre: Skipped 2 previous similar messages [ 1954.788514] LustreError: 107964:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1954.791410] LustreError: 107964:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1954.818351] Lustre: server umount lustre-MDT0000 complete [ 1956.210364] LustreError: 105609:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036445 with bad export cookie 6271119891199395271 [ 1956.214327] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1956.214497] LustreError: 105609:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1966.564813] LustreError: 108165:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1966.567348] LustreError: 108165:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1966.621495] Lustre: server umount lustre-MDT0001 complete [ 1969.628627] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 14:27:38 (1767036458) [ 1970.869861] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1976.620451] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_hostid [ 1979.829216] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 1983.504164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1985.763356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1987.807288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1989.970162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1992.363190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1992.387464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.470950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1992.482100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1992.517762] Lustre: lustre-MDT0000: new disk, initializing [ 1992.543676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1992.552364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1993.885425] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 1998.181094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1998.205037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1998.238825] Lustre: 111293:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1998.263989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1998.266510] Lustre: Skipped 1 previous similar message [ 1998.304172] Lustre: lustre-MDT0001: new disk, initializing [ 1998.351256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1998.354921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1999.720915] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2002.900237] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2004.120844] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2006.384446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2006.404936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2006.496551] Lustre: lustre-OST0000: new disk, initializing [ 2006.499071] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2006.526947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2006.529855] Lustre: Skipped 1 previous similar message [ 2008.502561] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2011.504458] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2012.667281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2012.671059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2012.705587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2012.880655] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2012.958035] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2014.348791] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2014.432811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2017.763554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.763808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2017.768498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2017.773902] Lustre: Skipped 1 previous similar message [ 2021.166239] LustreError: 114232:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2021.168635] LustreError: 114232:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2021.204570] Lustre: server umount lustre-OST0000 complete [ 2022.881445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.887221] Lustre: Skipped 1 previous similar message [ 2022.889948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2022.892927] Lustre: Skipped 1 previous similar message [ 2028.697944] Lustre: server umount lustre-MDT0000 complete [ 2029.973378] LustreError: 113040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036518 with bad export cookie 6271119891199396139 [ 2029.976831] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2029.978741] LustreError: 113040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2030.042752] LustreError: 114637:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2030.045827] LustreError: 114637:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2030.100759] Lustre: server umount lustre-MDT0001 complete [ 2033.107777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2033.267492] LustreError: 115112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2033.274551] LustreError: 115112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2033.296201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.687485] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2037.651823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2039.190124] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2041.101180] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2042.350053] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2044.685041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2046.736262] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2048.647888] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2061.613696] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2063.331688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.332447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2063.335711] Lustre: Skipped 1 previous similar message [ 2063.338048] LustreError: Skipped 1 previous similar message [ 2063.341639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2063.343584] Lustre: Skipped 3 previous similar messages [ 2064.687102] LustreError: 117475:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2064.689353] LustreError: 117475:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2064.727884] Lustre: server umount lustre-OST0000 complete [ 2068.182384] Lustre: server umount lustre-MDT0000 complete [ 2068.450384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2068.450810] LustreError: 115111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2068.453121] LustreError: Skipped 1 previous similar message [ 2068.460594] LustreError: 115111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2069.511446] LustreError: 115093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036558 with bad export cookie 6271119891199397021 [ 2069.515196] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2069.515919] LustreError: 115093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2069.636353] Lustre: server umount lustre-MDT0001 complete [ 2073.525220] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 2074.736588] Key type lgssc unregistered [ 2074.883686] LNet: 118408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2074.886683] LNetError: 118408:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2074.895482] LNet: Removed LNI 192.168.203.126@tcp [ 2075.231522] Key type .llcrypt unregistered [ 2075.232866] Key type ._llcrypt unregistered [ 2081.789841] Key type ._llcrypt registered [ 2081.792360] Key type .llcrypt registered [ 2081.839837] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_hostid [ 2088.363726] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 2088.797858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2088.839504] alg: No test for adler32 (adler32-zlib) [ 2089.704434] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2089.802101] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 2091.384178] Key type lgssc registered [ 2091.855733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2094.836945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2096.997185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2099.131050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2101.256880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2103.629931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2103.650397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2103.655534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.740262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2104.752436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2104.790807] Lustre: lustre-MDT0000: new disk, initializing [ 2104.818743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.825767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2106.148090] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2110.268975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2110.290966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2110.323050] Lustre: 121524:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2110.346724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2110.349428] Lustre: Skipped 1 previous similar message [ 2110.385789] Lustre: lustre-MDT0001: new disk, initializing [ 2110.419930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2110.431370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2110.435350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2111.738814] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2114.864938] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2116.030306] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2118.300200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2118.325273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2118.419158] Lustre: lustre-OST0000: new disk, initializing [ 2118.421050] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2118.453157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2120.340476] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2121.145489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2121.149580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2121.188477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2123.492616] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2124.859264] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2124.940917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2126.295741] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2126.385464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2129.070929] LustreError: 124471:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2129.106276] Lustre: server umount lustre-OST0000 complete [ 2130.912831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2130.915575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.920993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2131.425327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2131.425888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.545823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2136.548407] Lustre: Skipped 2 previous similar messages [ 2140.644318] LustreError: 124672:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2140.645939] LustreError: 124672:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2140.674542] Lustre: server umount lustre-MDT0000 complete [ 2141.941063] LustreError: 122305:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036630 with bad export cookie 800750860778090094 [ 2141.944384] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2141.944481] LustreError: 122305:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2151.396603] LustreError: 124873:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2151.398943] LustreError: 124873:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2151.447514] Lustre: server umount lustre-MDT0001 complete [ 2154.262948] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 14:30:43 (1767036643) [ 2156.467918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.632924] LustreError: 125545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2156.638661] LustreError: 125545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2156.657304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.966497] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2160.779483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2160.889609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2162.207806] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2163.997783] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2165.091760] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2165.730782] LustreError: 125574:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2165.733767] LustreError: 125574:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2165.736893] LustreError: 125574:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2166.241665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2166.241784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2166.242294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2166.245017] Lustre: Skipped 1 previous similar message [ 2170.337169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2170.339089] Lustre: Skipped 2 previous similar messages [ 2180.064130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2180.097400] LustreError: 126884:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2180.098969] LustreError: 126884:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2180.121215] Lustre: server umount lustre-MDT0000 complete [ 2180.577499] LustreError: 125540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2180.581269] LustreError: 125540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2181.264108] LustreError: 125526:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036670 with bad export cookie 800750860778090962 [ 2181.267672] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2181.268954] LustreError: 125526:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2181.379631] Lustre: server umount lustre-MDT0001 complete [ 2184.004657] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 14:31:13 (1767036673) [ 2186.067939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2201.504424] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2206.624391] LustreError: 127732:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.126@tcp: failed processing log, type 4: rc = -110 [ 2232.224274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2234.203447] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2236.052919] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2236.787415] LustreError: 128397:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2236.790026] LustreError: 128397:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2236.834763] Lustre: server umount lustre-OST0000 complete [ 2239.356296] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 14:32:08 (1767036728) [ 2241.517358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.672137] LustreError: 129067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.913421] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2245.684777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2247.046424] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2248.659360] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2249.827837] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2251.991930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2252.096302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.098552] Lustre: Skipped 2 previous similar messages [ 2253.920884] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2255.636416] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2260.450362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.451420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2260.454853] Lustre: Skipped 2 previous similar messages [ 2260.456097] Lustre: Skipped 3 previous similar messages [ 2264.061035] LustreError: 131315:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2264.062620] LustreError: 131315:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2264.085622] Lustre: server umount lustre-MDT0000 complete [ 2265.217152] LustreError: 130688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036754 with bad export cookie 800750860778091606 [ 2265.219748] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2265.222354] LustreError: 130688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2265.323151] Lustre: server umount lustre-MDT0001 complete [ 2276.693066] Lustre: server umount lustre-OST0000 complete [ 2279.286054] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 14:32:48 (1767036768) [ 2281.622401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2281.776675] LustreError: 132389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.981719] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2285.582164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2285.702637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2285.704314] Lustre: Skipped 1 previous similar message [ 2286.848832] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2288.667748] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2289.899128] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2292.205782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2294.157453] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2295.982555] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2297.368795] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2307.687975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2308.655121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec [ 2309.977285] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2310.059129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2311.136906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2311.138931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.144063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2311.145612] Lustre: Skipped 1 previous similar message [ 2313.188581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2316.780123] LustreError: 135209:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2316.783451] LustreError: 135209:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2316.819474] Lustre: server umount lustre-OST0000 complete [ 2324.179989] Lustre: server umount lustre-MDT0000 complete [ 2325.415186] LustreError: 132370:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036814 with bad export cookie 800750860778092663 [ 2325.419337] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2325.422661] LustreError: 132370:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2325.536290] Lustre: server umount lustre-MDT0001 complete [ 2328.282527] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 14:33:37 (1767036817) [ 2330.529539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2345.952563] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2351.072504] LustreError: 136255:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.126@tcp: failed processing log, type 4: rc = -110 [ 2376.672308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.674049] Lustre: Skipped 1 previous similar message [ 2378.426496] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2380.137757] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2382.321826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.472243] LustreError: 136283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.520399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2383.794178] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2386.676852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2388.071586] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2389.776061] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2391.018928] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2392.326582] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2392.397310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2393.773093] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2393.845596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2397.155679] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.156248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2397.156901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2397.156906] Lustre: Skipped 6 previous similar messages [ 2397.158671] Lustre: Skipped 4 previous similar messages [ 2400.635914] LustreError: 139030:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2400.638289] LustreError: 139030:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2400.662590] Lustre: server umount lustre-MDT0000 complete [ 2401.883186] LustreError: 136262:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036890 with bad export cookie 800750860778093671 [ 2401.886044] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2401.886169] LustreError: 136262:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2401.997298] Lustre: server umount lustre-MDT0001 complete [ 2413.533247] Lustre: server umount lustre-OST0000 complete [ 2416.142816] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 14:35:05 (1767036905) [ 2418.331618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2433.760613] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2438.880376] LustreError: 140078:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.126@tcp: failed processing log, type 4: rc = -110 [ 2466.361077] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2468.111279] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2470.306369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2470.451121] LustreError: 140105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.505338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2471.807672] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2474.707466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2476.155312] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2477.992252] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2479.272828] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2481.645662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2481.673114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2481.729626] Lustre: lustre-OST0001: new disk, initializing [ 2481.731668] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2483.788391] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2486.928392] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2487.793184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2487.796866] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2487.826045] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2488.370047] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2488.452760] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2489.858858] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2489.937969] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2490.848918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2490.852317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.857753] Lustre: Skipped 1 previous similar message [ 2490.860228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2490.862516] Lustre: Skipped 2 previous similar messages [ 2496.685365] LustreError: 143809:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2496.687701] LustreError: 143809:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2496.703934] Lustre: server umount lustre-OST0000 complete [ 2498.019351] LustreError: 141046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.026489] LustreError: 141046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2512.352077] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2512.403810] Lustre: server umount lustre-OST0001 complete [ 2515.937048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2515.939404] LustreError: Skipped 2 previous similar messages [ 2519.768852] Lustre: server umount lustre-MDT0000 complete [ 2521.041362] LustreError: 140085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037010 with bad export cookie 800750860778094616 [ 2521.044451] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2521.045842] LustreError: 140085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2521.167091] Lustre: server umount lustre-MDT0001 complete [ 2523.044280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2524.307385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2525.618517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2526.929484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2529.087473] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 14:36:58 (1767037018) [ 2529.627661] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2530.158206] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2530.692248] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 14:37:00 (1767037020) [ 2532.919976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.013080] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2533.023802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2533.090154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.092664] Lustre: Skipped 6 previous similar messages [ 2534.409921] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2537.322306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2537.356265] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2537.367939] Lustre: 146051:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2538.836381] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2540.661853] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2541.893116] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2544.175468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2544.242968] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2546.250097] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2548.036555] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2549.346487] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2552.858283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2553.516296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2554.853641] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2554.932685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2557.923785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2557.926649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.926906] LustreError: Skipped 1 previous similar message [ 2557.929648] Lustre: Skipped 5 previous similar messages [ 2561.673592] Lustre: server umount lustre-OST0000 complete [ 2575.288911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2577.325290] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2578.469876] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2578.475075] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2578.475107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2579.091402] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2580.582754] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2580.662484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2582.000470] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2582.073206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2583.282275] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa028c55f6800.ost_server_uuid 50 [ 2583.784598] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa028c55f6800.ost_server_uuid in FULL state after 0 sec [ 2583.844835] LustreError: 146056:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.203.26@tcp) returned error from blocking AST (req@00000000336232ca x1852872146996992 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000f94dfa28/0xb1cd6aa5244495d lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.26@tcp remote: 0xacd8074040dba8d expref: 5 pid: 146058 timeout: 2683 lvb_type: 0 [ 2583.860778] LustreError: lustre-MDT0000: A client on nid 192.168.203.26@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2583.865501] LustreError: 146047:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.26@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000f94dfa28/0xb1cd6aa5244495d lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.26@tcp remote: 0xacd8074040dba8d expref: 6 pid: 146058 timeout: 0 lvb_type: 0 [ 2590.990671] Lustre: server umount lustre-OST0000 complete [ 2598.362983] Lustre: server umount lustre-MDT0000 complete [ 2598.880913] LustreError: 146793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.884616] LustreError: 146793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2599.543396] LustreError: 146044:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037088 with bad export cookie 800750860778095792 [ 2599.546276] LustreError: 146044:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2599.546966] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2605.739843] Lustre: server umount lustre-MDT0001 complete [ 2609.305711] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 2610.358447] Key type lgssc unregistered [ 2610.479438] LNet: 151561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2610.481625] LNetError: 151561:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2610.490321] LNet: Removed LNI 192.168.203.126@tcp [ 2610.776324] Key type .llcrypt unregistered [ 2610.777676] Key type ._llcrypt unregistered [ 2619.379702] Key type ._llcrypt registered [ 2619.381377] Key type .llcrypt registered [ 2619.422691] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 2619.745953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2619.777029] alg: No test for adler32 (adler32-zlib) [ 2620.631906] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2620.715146] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 2622.296215] Key type lgssc registered [ 2622.671343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2625.249343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2625.254553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.408570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2627.673074] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2630.431350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2630.549219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2631.896843] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2633.684777] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2634.963648] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2637.199787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2637.315262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2639.146246] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2640.883326] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2651.690330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2658.611440] Lustre: Failing over lustre-MDT0000 [ 2658.757706] LustreError: 155200:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2658.786627] Lustre: server umount lustre-MDT0000 complete [ 2661.344791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2661.346687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.352751] LustreError: 153645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.356488] LustreError: 153645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2661.859845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.860675] LustreError: 154290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.865703] Lustre: Skipped 1 previous similar message [ 2666.977629] LustreError: 153645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.983444] LustreError: 153645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2669.671104] LustreError: 152914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.26@tcp arrived at 1767037158 with bad export cookie 295109780399783910 [ 2669.675818] LustreError: 152914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2672.017178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.054041] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2672.098255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2672.100200] Lustre: Skipped 2 previous similar messages [ 2672.143071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2672.174290] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2673.465665] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2674.176335] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2674.180405] Lustre: lustre-MDT0000: Denying connection for new client 45a50beb-c1e6-49fa-b133-2b1415780807 (at 192.168.203.26@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2677.227605] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2681.313107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2681.315347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.319876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2682.338502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2682.340136] Lustre: Skipped 1 previous similar message [ 2687.021145] LustreError: 156126:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2687.023351] LustreError: 156126:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2687.058091] Lustre: server umount lustre-OST0000 complete [ 2688.268096] LustreError: 156330:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2688.271387] Lustre: 155698:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2688.273741] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2688.279985] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2688.284116] LustreError: 155698:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000df1bb836 x1852872703642112/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2688.289343] LustreError: 155698:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2688.291942] LustreError: 155698:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2688.294697] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2688.376777] LustreError: 156330:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2688.378441] LustreError: 156330:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2688.411399] Lustre: server umount lustre-MDT0000 complete [ 2689.690314] LustreError: 152913:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037178 with bad export cookie 295109780399784260 [ 2689.693888] LustreError: 152913:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2689.694804] LustreError: 152241:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2689.695101] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2695.807927] LustreError: 156533:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2695.809726] LustreError: 156533:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2695.866537] Lustre: server umount lustre-MDT0001 complete [ 2699.430542] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 2700.431192] Key type lgssc unregistered [ 2700.564369] LNet: 157064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2700.566378] LNetError: 157064:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2700.577293] LNet: Removed LNI 192.168.203.126@tcp [ 2700.877194] Key type .llcrypt unregistered [ 2700.878178] Key type ._llcrypt unregistered [ 2709.979021] Key type ._llcrypt registered [ 2709.979935] Key type .llcrypt registered [ 2710.020543] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 2710.368795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2710.404084] alg: No test for adler32 (adler32-zlib) [ 2711.260965] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2711.342951] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 2712.928147] Key type lgssc registered [ 2713.276377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2715.801099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2715.808308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.970468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2718.284733] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2721.063348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2721.180527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2722.489318] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2724.282357] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2725.415340] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2727.662325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2727.777638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2729.690845] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2731.468525] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2741.095911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2746.339948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2746.340310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.340840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2746.350287] Lustre: Skipped 1 previous similar message [ 2749.230493] LustreError: 160700:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2749.261620] Lustre: server umount lustre-OST0000 complete [ 2751.457614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.458460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2751.460386] Lustre: Skipped 1 previous similar message [ 2751.461881] Lustre: Skipped 1 previous similar message [ 2756.577190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2756.579829] Lustre: Skipped 2 previous similar messages [ 2756.608399] LustreError: 160902:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2756.610151] LustreError: 160902:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2756.637585] Lustre: server umount lustre-MDT0000 complete [ 2757.849167] LustreError: 158414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037246 with bad export cookie 8063604828173783961 [ 2757.852192] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2757.853552] LustreError: 158414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2757.923879] LustreError: 161104:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2757.925567] LustreError: 161104:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2757.971978] Lustre: server umount lustre-MDT0001 complete [ 2761.512089] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 2762.577807] Key type lgssc unregistered [ 2762.696397] LNet: 161635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2762.699093] LNetError: 161635:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2762.708316] LNet: Removed LNI 192.168.203.126@tcp [ 2762.996480] Key type .llcrypt unregistered [ 2762.998238] Key type ._llcrypt unregistered [ 2774.119452] Key type ._llcrypt registered [ 2774.120339] Key type .llcrypt registered [ 2774.155323] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 2774.467570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2774.514920] alg: No test for adler32 (adler32-zlib) [ 2775.358540] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2775.433875] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 2777.016140] Key type lgssc registered [ 2777.368391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2779.863109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2779.868867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2781.015756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.251316] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2785.039049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2785.161186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2786.456228] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2788.226784] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2789.400631] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2791.551349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2791.659860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2793.525573] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2795.227310] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2805.991296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2816.484117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2816.484243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2816.487750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2816.488850] LustreError: Skipped 1 previous similar message [ 2818.479080] LustreError: 165563:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2818.513620] Lustre: server umount lustre-OST0000 complete [ 2821.089580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2821.092659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2821.098345] Lustre: Skipped 1 previous similar message [ 2821.101588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2821.103385] Lustre: Skipped 1 previous similar message [ 2826.721216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2826.722733] Lustre: Skipped 2 previous similar messages [ 2831.841493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2831.843083] Lustre: Skipped 1 previous similar message [ 2834.400110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2834.431992] LustreError: 165766:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2834.433945] LustreError: 165766:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2834.464535] Lustre: server umount lustre-MDT0000 complete [ 2835.711255] LustreError: 163273:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037324 with bad export cookie 684574281400205570 [ 2835.714270] LustreError: 163273:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2835.715383] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2835.777438] LustreError: 165968:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2835.779331] LustreError: 165968:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2835.833244] Lustre: server umount lustre-MDT0001 complete [ 2839.500742] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 2840.716189] Key type lgssc unregistered [ 2840.843575] LNet: 166499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2840.846981] LNetError: 166499:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2840.857420] LNet: Removed LNI 192.168.203.126@tcp [ 2841.175189] Key type .llcrypt unregistered [ 2841.176568] Key type ._llcrypt unregistered [ 2849.968215] Key type ._llcrypt registered [ 2849.969190] Key type .llcrypt registered [ 2850.007149] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 2850.395918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2850.410660] alg: No test for adler32 (adler32-zlib) [ 2851.254310] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2851.330889] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 2852.912112] Key type lgssc registered [ 2853.269519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2856.940893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2856.947588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2858.086337] Lustre: *** cfs_fail_loc=135, val=0*** [ 2858.129107] LustreError: 167987:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2858.131036] LustreError: 167987:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.126@tcp: cfg command failed: rc = -2 [ 2858.133369] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2858.133369] [ 2858.135621] LustreError: MGC192.168.203.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2858.139114] LustreError: 167944:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2858.141446] LustreError: 167944:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2858.143562] LustreError: 167944:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2858.145359] LustreError: 167944:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2858.148223] LustreError: 167944:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2858.181201] Lustre: server umount lustre-MDT0000 complete [ 2858.182601] LustreError: 167944:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2860.409186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2860.559986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2861.872899] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2864.718063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2864.846121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2866.192582] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2867.950372] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2869.153897] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2869.817437] LustreError: 168390:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2869.820531] LustreError: 168390:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2869.823046] LustreError: 168390:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2870.241825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2870.242353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2870.247935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2870.249679] Lustre: Skipped 1 previous similar message [ 2874.337576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2874.340779] Lustre: Skipped 1 previous similar message [ 2875.964079] LustreError: 169703:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2875.965752] LustreError: 169703:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2875.990227] Lustre: server umount lustre-MDT0000 complete [ 2877.209629] LustreError: 168342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037366 with bad export cookie 6265714404768300957 [ 2877.212861] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.213058] LustreError: 168342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2877.290570] LustreError: 169906:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2877.293136] LustreError: 169906:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2877.346681] Lustre: server umount lustre-MDT0001 complete [ 2880.764540] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 2881.823440] Key type lgssc unregistered [ 2881.963393] LNet: 170437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2881.965911] LNetError: 170437:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2881.972298] LNet: Removed LNI 192.168.203.126@tcp [ 2882.284809] Key type .llcrypt unregistered [ 2882.285723] Key type ._llcrypt unregistered [ 2889.293168] Key type ._llcrypt registered [ 2889.294785] Key type .llcrypt registered [ 2889.338445] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 2889.437261] Key type .llcrypt unregistered [ 2889.438403] Key type ._llcrypt unregistered [ 2896.475334] Key type ._llcrypt registered [ 2896.477042] Key type .llcrypt registered [ 2896.517166] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 2896.932865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2896.939565] alg: No test for adler32 (adler32-zlib) [ 2897.811131] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2897.892083] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 2899.472098] Key type lgssc registered [ 2899.829839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2902.538115] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2902.546986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2917.984352] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2923.104201] LustreError: 172504:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.126@tcp: failed processing log, type 4: rc = -110 [ 2949.728217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.644223] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2953.403688] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2955.624970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.789529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2955.816159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2957.066036] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2959.966430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2960.113408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2961.492348] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 2963.335123] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2964.510349] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2965.436261] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2969.573121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.574091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2969.576751] Lustre: Skipped 1 previous similar message [ 2972.463163] LustreError: 174984:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2972.500346] Lustre: server umount lustre-OST0000 complete [ 2974.690133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.690947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.696824] Lustre: Skipped 1 previous similar message [ 2974.699458] Lustre: Skipped 2 previous similar messages [ 2979.809280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2979.812179] Lustre: Skipped 1 previous similar message [ 2979.904077] LustreError: 175186:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2979.905869] LustreError: 175186:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2979.931937] Lustre: server umount lustre-MDT0000 complete [ 2981.126727] LustreError: 172511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037470 with bad export cookie 16709991788668745849 [ 2981.129652] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2981.130478] LustreError: 172511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2981.195313] LustreError: 175388:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2981.198432] LustreError: 175388:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2981.251194] Lustre: server umount lustre-MDT0001 complete [ 2984.655641] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 2985.673081] Key type lgssc unregistered [ 2985.805353] LNet: 175919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2985.808631] LNetError: 175919:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2985.816318] LNet: Removed LNI 192.168.203.126@tcp [ 2986.147080] Key type .llcrypt unregistered [ 2986.148911] Key type ._llcrypt unregistered [ 2995.109400] Key type ._llcrypt registered [ 2995.111073] Key type .llcrypt registered [ 2995.151232] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 2995.496972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2995.543492] alg: No test for adler32 (adler32-zlib) [ 2996.420912] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2996.532500] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 2998.120177] Key type lgssc registered [ 2998.511681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3001.194746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3001.201701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.357353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.660099] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3006.464528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3006.609207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3007.988018] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3009.708214] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3010.841354] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3013.045989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3013.092094] LustreError: 178886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.096168] LustreError: 178886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3013.159244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.705077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3014.950548] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3016.697413] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3018.759860] Lustre: Failing over lustre-MDT0000 [ 3018.830450] LustreError: 179584:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3018.858233] Lustre: server umount lustre-MDT0000 complete [ 3021.796477] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.797970] LustreError: 177287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3021.799906] Lustre: Skipped 2 previous similar messages [ 3021.806558] LustreError: 177287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3023.078792] LustreError: 177283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.914310] LustreError: 177283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.919643] LustreError: 177283:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3031.726323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.763393] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3031.849776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.880905] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3033.178048] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3033.311583] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3034.672858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3037.157567] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3037.164519] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3037.180465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3039.149790] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3044.834136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3044.836044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.839871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3047.396731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3047.397100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3047.397562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3047.408466] Lustre: Skipped 1 previous similar message [ 3050.414289] LustreError: 180861:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3050.416112] LustreError: 180861:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3050.450786] Lustre: server umount lustre-OST0000 complete [ 3052.513975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3052.514188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3052.520043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3052.521341] Lustre: Skipped 1 previous similar message [ 3057.633084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3057.635717] Lustre: Skipped 1 previous similar message [ 3057.723031] LustreError: 181066:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3057.724744] LustreError: 181066:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3057.747959] Lustre: server umount lustre-MDT0000 complete [ 3058.956313] LustreError: 177269:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037547 with bad export cookie 1933368201847373470 [ 3058.959639] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3058.960508] LustreError: 177269:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3059.083269] Lustre: server umount lustre-MDT0001 complete [ 3062.599233] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 3063.554271] Key type lgssc unregistered [ 3063.681393] LNet: 181798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3063.683266] LNetError: 181798:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3063.691285] LNet: Removed LNI 192.168.203.126@tcp [ 3063.968659] Key type .llcrypt unregistered [ 3063.969674] Key type ._llcrypt unregistered [ 3072.734190] Key type ._llcrypt registered [ 3072.735261] Key type .llcrypt registered [ 3072.771333] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 3073.114571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3073.144335] alg: No test for adler32 (adler32-zlib) [ 3073.996298] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 3074.078831] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 3075.656190] Key type lgssc registered [ 3076.016381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3078.507453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3078.513419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.666063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.956403] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3083.887640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3084.013499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3085.225584] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3086.943505] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3088.077925] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3090.277153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3090.390916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3092.324924] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3094.114869] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3104.745027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3110.161839] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3111.566017] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3120.597129] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3130.342793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.343120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3130.344359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3130.349082] Lustre: Skipped 1 previous similar message [ 3135.459872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3135.461925] Lustre: Skipped 2 previous similar messages [ 3135.854070] LustreError: 185602:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3135.894342] Lustre: server umount lustre-OST0000 complete [ 3140.577954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3140.578227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.580376] LustreError: Skipped 1 previous similar message [ 3140.584620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3140.586869] Lustre: Skipped 1 previous similar message [ 3143.233299] LustreError: 185803:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3143.236686] LustreError: 185803:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3143.265230] Lustre: server umount lustre-MDT0000 complete [ 3144.484296] LustreError: 183148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037633 with bad export cookie 8211473633379693159 [ 3144.487312] LustreError: 183148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3144.488445] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3144.557227] LustreError: 186004:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3144.558885] LustreError: 186004:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3144.617290] Lustre: server umount lustre-MDT0001 complete [ 3148.250880] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 3149.235281] Key type lgssc unregistered [ 3149.365382] LNet: 186535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3149.367325] LNetError: 186535:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3149.375316] LNet: Removed LNI 192.168.203.126@tcp [ 3149.653670] Key type .llcrypt unregistered [ 3149.654957] Key type ._llcrypt unregistered [ 3158.808075] Key type ._llcrypt registered [ 3158.809113] Key type .llcrypt registered [ 3158.852924] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 3159.299538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3159.336932] alg: No test for adler32 (adler32-zlib) [ 3160.182881] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 3160.263169] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 3161.840185] Key type lgssc registered [ 3162.188184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3164.738927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3164.745497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3165.894783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.151865] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3169.884307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3170.018317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3171.313286] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3173.093108] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3174.261112] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3176.584739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3176.722600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3178.602547] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3180.424276] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3190.056067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3192.358749] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3203.727163] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3210.430935] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3218.465307] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3227.104933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3227.108037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.111988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3231.206222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3231.206900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3231.210687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3232.493078] LustreError: 192144:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3232.526093] Lustre: server umount lustre-OST0000 complete [ 3236.321722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3236.326136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3236.327929] Lustre: Skipped 2 previous similar messages [ 3241.441635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3241.443625] Lustre: Skipped 1 previous similar message [ 3246.561567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3246.563220] Lustre: Skipped 1 previous similar message [ 3248.096180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3248.138464] LustreError: 192345:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3248.140600] LustreError: 192345:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3248.165967] Lustre: server umount lustre-MDT0000 complete [ 3249.418724] LustreError: 187885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037738 with bad export cookie 16484247606497709375 [ 3249.423578] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3249.424127] LustreError: 187885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3249.504628] LustreError: 192548:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3249.506297] LustreError: 192548:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3249.566891] Lustre: server umount lustre-MDT0001 complete [ 3253.092756] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 3254.120873] Key type lgssc unregistered [ 3254.257520] LNet: 193078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3254.259715] LNetError: 193078:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3254.272451] LNet: Removed LNI 192.168.203.126@tcp [ 3254.600506] Key type .llcrypt unregistered [ 3254.601462] Key type ._llcrypt unregistered [ 3263.739704] Key type ._llcrypt registered [ 3263.740741] Key type .llcrypt registered [ 3263.779405] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 3264.188855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3264.223764] alg: No test for adler32 (adler32-zlib) [ 3265.090290] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 3265.181775] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 3266.776143] Key type lgssc registered [ 3267.211275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3269.972225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3269.979592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3271.173538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3272.616791] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3275.658054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3275.815799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3277.217201] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3278.992813] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3280.301373] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3282.669614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3282.817338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3284.704761] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3286.473896] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3296.173186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3301.346718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3301.348555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3301.349560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3301.351725] Lustre: Skipped 1 previous similar message [ 3301.354509] Lustre: Skipped 1 previous similar message [ 3306.468238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3306.470239] Lustre: Skipped 1 previous similar message [ 3307.310725] LustreError: 196864:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3307.350244] Lustre: server umount lustre-OST0000 complete [ 3311.586792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3311.587852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3311.588544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3311.592138] LustreError: Skipped 1 previous similar message [ 3311.602572] Lustre: Skipped 1 previous similar message [ 3314.828417] LustreError: 197067:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3314.831623] LustreError: 197067:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3314.863740] Lustre: server umount lustre-MDT0000 complete [ 3316.206479] LustreError: 194431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037805 with bad export cookie 9769982734482963022 [ 3316.210086] LustreError: 194431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3316.211201] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3316.297641] LustreError: 197272:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3316.299739] LustreError: 197272:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3316.357288] Lustre: server umount lustre-MDT0001 complete [ 3319.922016] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 3321.197524] Key type lgssc unregistered [ 3321.365463] LNet: 197803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3321.368775] LNetError: 197803:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3321.378564] LNet: Removed LNI 192.168.203.126@tcp [ 3321.802991] Key type .llcrypt unregistered [ 3321.806147] Key type ._llcrypt unregistered [ 3329.413237] Key type ._llcrypt registered [ 3329.414755] Key type .llcrypt registered [ 3329.453100] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 3329.905431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3329.948957] alg: No test for adler32 (adler32-zlib) [ 3330.815017] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 3330.908287] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 3332.496242] Key type lgssc registered [ 3332.864374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3335.544796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3335.550917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3336.735790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3338.081109] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3341.023969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3341.165250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3342.464210] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3344.261202] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3345.434245] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3347.755820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3347.895408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3349.886786] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3351.783912] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3361.192331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3366.370373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3366.372118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.372810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3366.374909] Lustre: Skipped 1 previous similar message [ 3366.377247] Lustre: Skipped 1 previous similar message [ 3367.798174] LustreError: 201439:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3367.852599] Lustre: server umount lustre-OST0000 complete [ 3371.325836] LustreError: 201640:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3371.327453] LustreError: 201640:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3371.357978] Lustre: server umount lustre-MDT0000 complete [ 3371.490823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3371.491693] LustreError: 199020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3371.495139] Lustre: Skipped 1 previous similar message [ 3371.501029] LustreError: 199020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3372.770736] LustreError: 199006:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037861 with bad export cookie 15964765209557158392 [ 3372.778310] LustreError: 199006:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3372.778621] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3372.864741] LustreError: 201842:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3372.866375] LustreError: 201842:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3372.940249] Lustre: server umount lustre-MDT0001 complete [ 3376.969919] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 3378.270439] Key type lgssc unregistered [ 3378.407451] LNet: 202373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3378.410354] LNetError: 202373:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3378.416569] LNet: Removed LNI 192.168.203.126@tcp [ 3378.876500] Key type .llcrypt unregistered [ 3378.877929] Key type ._llcrypt unregistered [ 3387.157214] Key type ._llcrypt registered [ 3387.159026] Key type .llcrypt registered [ 3387.220546] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 3387.517912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3387.524564] alg: No test for adler32 (adler32-zlib) [ 3388.379588] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 3388.469978] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 3390.056199] Key type lgssc registered [ 3390.463226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3393.303194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3393.308524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.488207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3395.802784] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3399.277433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3399.424169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3400.789119] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3402.828560] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3404.251385] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3407.064713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3407.199163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3409.167734] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3410.952158] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3419.503443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3424.528521] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3424.567874] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3424.732404] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3427.297947] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3429.302956] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3429.880741] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3440.438699] Lustre: Permanently deactivating lustre-OST0001 [ 3440.444633] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3446.262724] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3449.224920] Lustre: Permanently reactivating lustre-OST0001 [ 3449.231602] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3449.233344] Lustre: Skipped 2 previous similar messages [ 3452.533937] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3452.537554] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3452.539512] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3452.542421] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3452.552887] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3462.628272] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.629977] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3462.630983] Lustre: Skipped 2 previous similar messages [ 3462.633133] Lustre: Skipped 1 previous similar message [ 3465.389124] LustreError: 207062:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3465.404377] Lustre: server umount lustre-OST0001 complete [ 3467.747769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3467.748233] LustreError: 205561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3467.749287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3467.750579] Lustre: Skipped 1 previous similar message [ 3467.753887] LustreError: 205561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3472.687079] LustreError: 207263:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3472.688713] LustreError: 207263:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3472.726269] Lustre: server umount lustre-OST0000 complete [ 3476.449029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3476.452814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3476.457334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3476.458988] Lustre: Skipped 1 previous similar message [ 3483.106379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3483.111547] Lustre: Skipped 3 previous similar messages [ 3487.202521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3487.205866] Lustre: Skipped 1 previous similar message [ 3488.224125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3488.259907] LustreError: 207464:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3488.261448] LustreError: 207464:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3488.286335] Lustre: server umount lustre-MDT0000 complete [ 3489.633107] LustreError: 206123:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037978 with bad export cookie 7726059224060671459 [ 3489.638835] LustreError: 206123:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3489.639404] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3489.785054] Lustre: server umount lustre-MDT0001 complete [ 3493.643555] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 3494.983372] Key type lgssc unregistered [ 3495.155660] LNet: 208198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3495.159964] LNetError: 208198:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3495.169587] LNet: Removed LNI 192.168.203.126@tcp [ 3495.612685] Key type .llcrypt unregistered [ 3495.614649] Key type ._llcrypt unregistered [ 3508.456952] Key type ._llcrypt registered [ 3508.457887] Key type .llcrypt registered [ 3508.492207] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_hostid [ 3514.371890] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 3514.835353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3514.851762] alg: No test for adler32 (adler32-zlib) [ 3515.698185] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 3515.783754] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 3517.368223] Key type lgssc registered [ 3517.833959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3521.816923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3524.755114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3527.271943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3530.253589] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3533.268290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3533.300667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3533.308974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3534.426087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3534.445603] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3534.499751] Lustre: lustre-MDT0000: new disk, initializing [ 3534.538752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3534.549186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3536.143212] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3541.307249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3541.346748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3541.394317] Lustre: 212117:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3541.421913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3541.424299] Lustre: Skipped 1 previous similar message [ 3541.465632] Lustre: lustre-MDT0001: new disk, initializing [ 3541.507857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3541.522896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3541.526441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3543.172603] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3546.944164] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3548.431699] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3550.716610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3550.737180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3550.842540] Lustre: lustre-OST0000: new disk, initializing [ 3550.844154] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3550.872585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3552.567357] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3555.423867] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3556.609658] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3556.852545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3556.856148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3556.886434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3557.684365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3558.844196] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3558.904437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3561.953634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3561.955371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3561.956974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3561.957827] Lustre: Skipped 1 previous similar message [ 3565.549098] LustreError: 215064:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3565.582194] Lustre: server umount lustre-OST0000 complete [ 3567.073940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3567.074262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3567.074703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3567.074707] Lustre: Skipped 1 previous similar message [ 3567.075724] LustreError: Skipped 1 previous similar message [ 3567.078570] Lustre: Skipped 1 previous similar message [ 3572.193078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3572.194918] Lustre: Skipped 2 previous similar messages [ 3572.802881] LustreError: 215265:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3572.804448] LustreError: 215265:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3572.830578] Lustre: server umount lustre-MDT0000 complete [ 3573.959239] LustreError: 212108:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038062 with bad export cookie 9992872481786453037 [ 3573.962596] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3573.967538] LustreError: 212108:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3574.033600] LustreError: 215466:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3574.036442] LustreError: 215466:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3574.089850] Lustre: server umount lustre-MDT0001 complete [ 3576.551493] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 14:54:25 (1767038065) [ 3578.583325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3578.750243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3579.870863] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3582.400012] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3583.665298] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3585.224213] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3586.283768] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3588.338848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3588.449375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3588.450935] Lustre: Skipped 1 previous similar message [ 3590.128252] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3591.705778] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3593.204852] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3601.937349] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3608.718341] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3617.579701] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3626.348272] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3642.860867] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3642.864365] Lustre: Skipped 1 previous similar message [ 3683.829240] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3683.831290] Lustre: Skipped 4 previous similar messages [ 3754.166302] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3754.170830] Lustre: Skipped 8 previous similar messages [ 3763.682926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3763.687354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.695456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3766.758491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3766.759040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3768.556108] LustreError: 219625:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3768.558158] LustreError: 219625:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3768.589268] Lustre: server umount lustre-OST0000 complete [ 3771.874694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3771.874951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.883512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3771.885541] Lustre: Skipped 3 previous similar messages [ 3782.113045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3782.114475] Lustre: Skipped 3 previous similar messages [ 3784.160078] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3784.196470] LustreError: 219831:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3784.199433] LustreError: 219831:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3784.225974] Lustre: server umount lustre-MDT0000 complete [ 3785.421331] LustreError: 216118:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038274 with bad export cookie 9992872481786453919 [ 3785.423652] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3785.424030] LustreError: 216118:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3785.536781] Lustre: server umount lustre-MDT0001 complete [ 3788.846038] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 3789.811462] Key type lgssc unregistered [ 3789.926304] LNet: 220567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3789.928196] LNetError: 220567:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3789.934269] LNet: Removed LNI 192.168.203.126@tcp [ 3790.223972] Key type .llcrypt unregistered [ 3790.225335] Key type ._llcrypt unregistered [ 3798.374090] Key type ._llcrypt registered [ 3798.374959] Key type .llcrypt registered [ 3798.406978] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 3798.694040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3798.707979] alg: No test for adler32 (adler32-zlib) [ 3799.556470] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 3799.636959] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 3801.224103] Key type lgssc registered [ 3801.670774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3805.058245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3805.069394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.323648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3807.964616] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3812.038579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3812.270204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3814.187305] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3816.834250] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3818.555337] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3821.721352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3821.846483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3824.493316] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug -1 all [ 3827.129747] Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3836.389283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.393585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3836.393785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3836.396903] Lustre: Skipped 1 previous similar message [ 3841.509023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3841.511751] Lustre: Skipped 2 previous similar messages [ 3842.164293] LustreError: 224342:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3842.230904] Lustre: server umount lustre-OST0000 complete [ 3846.627863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.630106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3846.635198] Lustre: Skipped 1 previous similar message [ 3851.746738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3851.750308] Lustre: Skipped 2 previous similar messages [ 3856.866411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3856.869752] Lustre: Skipped 1 previous similar message [ 3858.400177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3858.452890] LustreError: 224547:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3858.456089] LustreError: 224547:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3858.489469] Lustre: server umount lustre-MDT0000 complete [ 3860.272670] LustreError: 223561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038349 with bad export cookie 2104494826461981741 [ 3860.276521] LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3860.279869] LustreError: 223561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3860.371968] LustreError: 224749:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3860.374119] LustreError: 224749:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3860.434096] Lustre: server umount lustre-MDT0001 complete [ 3864.998741] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 3866.510393] Key type lgssc unregistered [ 3866.691624] LNet: 225280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3866.696446] LNetError: 225280:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3866.709699] LNet: Removed LNI 192.168.203.126@tcp [ 3867.157662] Key type .llcrypt unregistered [ 3867.158783] Key type ._llcrypt unregistered [ 3877.450692] Key type ._llcrypt registered [ 3877.453063] Key type .llcrypt registered [ 3877.501224] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 3877.791759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3877.800057] alg: No test for adler32 (adler32-zlib) [ 3878.657465] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 3878.731759] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 3880.304206] Key type lgssc registered [ 3880.878313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4145.268775] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 4146.359707] Key type lgssc unregistered [ 4146.468338] LNet: 227250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4146.471171] LNetError: 227250:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4146.478316] LNet: Removed LNI 192.168.203.126@tcp [ 4146.733715] Key type .llcrypt unregistered [ 4146.734715] Key type ._llcrypt unregistered [ 4156.324164] Key type ._llcrypt registered [ 4156.325055] Key type .llcrypt registered [ 4156.358885] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_hostid [ 4164.029765] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing load_modules_local [ 4164.352187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4164.410072] alg: No test for adler32 (adler32-zlib) [ 4165.262318] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 4165.339799] LNet: Added LNI 192.168.203.126@tcp [8/256/0/180] [ 4166.920224] Key type lgssc registered [ 4167.538922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4172.053904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4175.415429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4178.530595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4181.357574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4181.854500] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4030 sec ========== 15:04:31 (1767038671) [ 4182.315412] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:04:31 (1767038671) === [ 4183.267689] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:04:32 (1767038672) === [ 4194.936492] Lustre: DEBUG MARKER: oleg326-server.virtnet: executing unload_modules_local [ 4196.206764] Key type lgssc unregistered [ 4196.342465] LNet: 232412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4196.346943] LNetError: 232412:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4196.358355] LNet: Removed LNI 192.168.203.126@tcp [ 4196.703556] Key type .llcrypt unregistered [ 4196.705602] Key type ._llcrypt unregistered