[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 546701205 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2512MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003341] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005020] kvm-guest: setup PV IPIs [ 0.008379] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010017] pid_max: default: 32768 minimum: 301 [ 0.011238] LSM: Security Framework initializing [ 0.012092] Yama: becoming mindful. [ 0.013061] SELinux: Initializing. [ 0.014114] *** VALIDATE selinux *** [ 0.024333] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030236] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031203] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032168] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033144] *** VALIDATE tmpfs *** [ 0.034000] *** VALIDATE proc *** [ 0.035114] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037321] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038182] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040052] Spectre V2 : User space: Vulnerable [ 0.041014] Speculative Store Bypass: Vulnerable [ 0.043000] debug: unmapping init [mem 0xffffffff9ba59000-0xffffffff9ba60fff] [ 0.044195] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045753] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046030] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050026] ... max period: 00007fffffffffff [ 0.051010] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053324] rcu: Hierarchical SRCU implementation. [ 0.055686] smp: Bringing up secondary CPUs ... [ 0.057512] x86: Booting SMP configuration: [ 0.058035] .... node #0, CPUs: #1 #2 #3 [ 0.065169] smp: Brought up 1 node, 4 CPUs [ 0.067013] smpboot: Max logical packages: 1 [ 0.068013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.109541] node 0 deferred pages initialised in 36ms [ 0.115140] devtmpfs: initialized [ 0.116295] x86/mm: Memory block size: 128MB [ 0.120048] gcov: version magic: 0x41383552 [ 0.125378] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.129099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.132332] pinctrl core: initialized pinctrl subsystem [ 0.134304] [ 0.134869] ************************************************************* [ 0.138017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.139000] ** ** [ 0.140010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142009] ** ** [ 0.145014] ** This means that this kernel is built to expose internal ** [ 0.147008] ** IOMMU data structures, which may compromise security on ** [ 0.150066] ** your system. ** [ 0.152012] ** ** [ 0.154121] ** If you see this message and you are not debugging the ** [ 0.157012] ** kernel, report this immediately to your vendor! ** [ 0.159012] ** ** [ 0.162053] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165011] ************************************************************* [ 0.167968] NET: Registered protocol family 16 [ 0.170534] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.181209] cpuidle: using governor menu [ 0.182930] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185629] PCI: Using configuration type 1 for base access [ 0.188144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.203672] cryptd: max_cpu_qlen set to 1000 [ 0.208231] ACPI: Added _OSI(Module Device) [ 0.209017] ACPI: Added _OSI(Processor Device) [ 0.210027] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.212013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.224681] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242333] ACPI: Interpreter enabled [ 0.243000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243000] ACPI: Using IOAPIC for interrupt routing [ 0.244158] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245422] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.256049] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.258037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.263084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269732] acpiphp: Slot [2] registered [ 0.271134] acpiphp: Slot [3] registered [ 0.272108] acpiphp: Slot [4] registered [ 0.274089] acpiphp: Slot [5] registered [ 0.275179] acpiphp: Slot [6] registered [ 0.277119] acpiphp: Slot [7] registered [ 0.278111] acpiphp: Slot [8] registered [ 0.280088] acpiphp: Slot [9] registered [ 0.281127] acpiphp: Slot [10] registered [ 0.284206] acpiphp: Slot [11] registered [ 0.286107] acpiphp: Slot [12] registered [ 0.287135] acpiphp: Slot [13] registered [ 0.289076] acpiphp: Slot [14] registered [ 0.290115] acpiphp: Slot [15] registered [ 0.291091] acpiphp: Slot [16] registered [ 0.293075] acpiphp: Slot [17] registered [ 0.294115] acpiphp: Slot [18] registered [ 0.295015] acpiphp: Slot [19] registered [ 0.296082] acpiphp: Slot [20] registered [ 0.298111] acpiphp: Slot [21] registered [ 0.300089] acpiphp: Slot [22] registered [ 0.301105] acpiphp: Slot [23] registered [ 0.303104] acpiphp: Slot [24] registered [ 0.305234] acpiphp: Slot [25] registered [ 0.307122] acpiphp: Slot [26] registered [ 0.310128] acpiphp: Slot [27] registered [ 0.311123] acpiphp: Slot [28] registered [ 0.315295] acpiphp: Slot [29] registered [ 0.317190] acpiphp: Slot [30] registered [ 0.321155] acpiphp: Slot [31] registered [ 0.322087] PCI host bridge to bus 0000:00 [ 0.325029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.326024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.327000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.329026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333027] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.337035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.339177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.346248] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352799] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.372016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.375000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.375015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.378014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.381019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.384521] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.395822] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.399042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.406543] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.414018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.426023] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.435021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.440014] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.445021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.452023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.475019] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.492000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.505022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.516018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.535000] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.548420] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.558017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.569019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.586015] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.603356] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.608324] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.616016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.637016] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.648462] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.655018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.661013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.680053] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.690731] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.708013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.714019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.734026] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.746310] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.747350] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.748000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.748296] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.749000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.752000] iommu: Default domain type: Passthrough [ 0.752000] SCSI subsystem initialized [ 0.755308] ACPI: bus type USB registered [ 0.756120] usbcore: registered new interface driver usbfs [ 0.758074] usbcore: registered new interface driver hub [ 0.759073] usbcore: registered new device driver usb [ 0.760135] pps_core: LinuxPPS API ver. 1 registered [ 0.761000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.761000] PTP clock support registered [ 0.761168] EDAC MC: Ver: 3.0.0 [ 0.762187] PCI: Using ACPI for IRQ routing [ 0.765501] NetLabel: Initializing [ 0.768017] NetLabel: domain hash size = 128 [ 0.770016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.773144] NetLabel: unlabeled traffic allowed by default [ 0.776344] vgaarb: loaded [ 0.778075] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.779008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.785021] clocksource: Switched to clocksource kvm-clock [ 0.937271] VFS: Disk quotas dquot_6.6.0 [ 0.942280] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.947205] *** VALIDATE ramfs *** [ 0.950036] *** VALIDATE hugetlbfs *** [ 0.951497] pnp: PnP ACPI init [ 0.954463] pnp: PnP ACPI: found 6 devices [ 0.972093] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.975458] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.977307] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.979361] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.981884] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.984444] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.987324] NET: Registered protocol family 2 [ 0.989927] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.996203] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.999918] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.005758] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.009391] TCP: Hash tables configured (established 65536 bind 65536) [ 1.012313] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.015475] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.018394] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.021541] NET: Registered protocol family 1 [ 1.025835] RPC: Registered named UNIX socket transport module. [ 1.028887] RPC: Registered udp transport module. [ 1.030270] RPC: Registered tcp transport module. [ 1.032111] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.035350] NET: Registered protocol family 44 [ 1.037068] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.039419] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.041713] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.044131] PCI: CLS 0 bytes, default 64 [ 1.046653] Unpacking initramfs... [ 2.765863] debug: unmapping init [mem 0xffff9e60fcc54000-0xffff9e60fffbffff] [ 2.770137] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.772360] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.775265] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.366985] Initialise system trusted keyrings [ 3.368466] Key type blacklist registered [ 3.370017] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.378266] zbud: loaded [ 3.380839] *** VALIDATE nfs *** [ 3.381801] *** VALIDATE nfs4 *** [ 3.382917] pstore: using deflate compression [ 3.386126] Platform Keyring initialized [ 3.513472] NET: Registered protocol family 38 [ 3.514786] Key type asymmetric registered [ 3.515914] Asymmetric key parser 'x509' registered [ 3.529992] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.535642] io scheduler mq-deadline registered [ 3.536903] io scheduler kyber registered [ 3.538103] io scheduler bfq registered [ 3.539868] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.542395] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.544926] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.547434] ACPI: Power Button [PWRF] [ 3.656837] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.761039] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.065876] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.201557] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.452182] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.482779] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.517403] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.521970] Non-volatile memory driver v1.3 [ 4.523451] Linux agpgart interface v0.103 [ 4.583461] virtio_blk virtio1: [vda] 133040 512-byte logical blocks (68.1 MB/65.0 MiB) [ 4.590933] vda: detected capacity change from 0 to 68116480 [ 4.609842] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.616029] vdb: detected capacity change from 0 to 1073741824 [ 4.638461] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.645481] vdc: detected capacity change from 0 to 2621440000 [ 4.672072] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.675039] vdd: detected capacity change from 0 to 2621440000 [ 4.697733] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.700716] vde: detected capacity change from 0 to 4294967296 [ 4.719818] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.729026] vdf: detected capacity change from 0 to 4294967296 [ 4.741221] libphy: Fixed MDIO Bus: probed [ 4.749750] usbcore: registered new interface driver usbserial_generic [ 4.754411] usbserial: USB Serial support registered for generic [ 4.756623] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.763827] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.765438] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.771502] mousedev: PS/2 mouse device common for all mice [ 4.774374] rtc_cmos 00:05: RTC can wake from S4 [ 4.778646] rtc_cmos 00:05: registered as rtc0 [ 4.780551] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.785408] intel_pstate: CPU model not supported [ 4.789165] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.806558] hid: raw HID events driver (C) Jiri Kosina [ 4.821269] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.827506] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.838486] usbcore: registered new interface driver usbhid [ 4.843012] usbhid: USB HID core driver [ 4.844449] drop_monitor: Initializing network drop monitor service [ 4.847281] Initializing XFRM netlink socket [ 4.850430] NET: Registered protocol family 10 [ 4.857791] Segment Routing with IPv6 [ 4.859215] NET: Registered protocol family 17 [ 4.862023] mpls_gso: MPLS GSO support [ 4.871570] RAS: Correctable Errors collector initialized. [ 4.873524] AVX version of gcm_enc/dec engaged. [ 4.875357] AES CTR mode by8 optimization enabled [ 5.079228] sched_clock: Marking stable (5079202361, 0)->(6247098641, -1167896280) [ 5.085538] registered taskstats version 1 [ 5.087323] Loading compiled-in X.509 certificates [ 5.090412] zswap: loaded using pool lzo/zbud [ 5.141242] Key type big_key registered [ 5.174399] Key type encrypted registered [ 5.175637] ima: No TPM chip found, activating TPM-bypass! [ 5.177218] ima: Allocated hash algorithm: sha1 [ 5.178393] ima: No architecture policies found [ 5.179637] evm: Initialising EVM extended attributes: [ 5.181119] evm: security.selinux [ 5.181980] evm: security.ima [ 5.186554] evm: security.capability [ 5.187800] evm: HMAC attrs: 0x1 [ 5.191321] rtc_cmos 00:05: setting system clock to 2025-08-31 21:30:14 UTC (1756675814) [ 5.197523] debug: unmapping init [mem 0xffffffff9ca03000-0xffffffff9cbfffff] [ 5.201474] debug: unmapping init [mem 0xffffffff9b782000-0xffffffff9ba58fff] [ 5.214733] Write protecting the kernel read-only data: 28672k [ 5.218322] debug: unmapping init [mem 0xffffffff99e03000-0xffffffff99ffffff] [ 5.220513] debug: unmapping init [mem 0xffffffff9a714000-0xffffffff9a7fffff] [ 5.280694] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.294422] systemd[1]: Detected virtualization kvm. [ 5.296597] systemd[1]: Detected architecture x86-64. [ 5.298219] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.342167] systemd[1]: No hostname configured. [ 5.344183] systemd[1]: Set hostname to . [ 5.346045] random: systemd: uninitialized urandom read (16 bytes read) [ 5.352134] systemd[1]: Initializing machine ID from random generator. [ 5.461427] random: ln: uninitialized urandom read (6 bytes read) [ 5.687457] random: systemd: uninitialized urandom read (16 bytes read) [ 5.692863] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.718983] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.730410] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.080957] device-mapper: uevent: version 1.0.3 [ 7.084581] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.754536] virtio_net virtio0 ens2: renamed from eth0 [ 8.820634] random: fast init done [ 9.121570] scsi host0: ata_piix [ 9.235336] scsi host1: ata_piix [ 9.240879] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.244257] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.697160] random: crng init done [ 13.701582] random: 7 urandom warning(s) missed due to ratelimiting [ 16.123732] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.289821] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.435668] printk: systemd: 25 output lines suppressed due to ratelimiting [ 19.977935] SELinux: Disabled at runtime. [ 20.057721] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.071481] systemd[1]: Detected virtualization kvm. [ 20.074062] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.418675] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.425877] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.433073] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.437831] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.445948] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.466093] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.489329] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ 21.786918] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 22.735057] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.471484] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.505688] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.962487] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.317979] EDAC sbridge: Ver: 1.1.2 [ 28.295765] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 28.652326] NFS: Registering the id_resolver key type [ 28.654159] Key type id_resolver registered [ 28.655712] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg357-server login: [ 56.264901] libcfs: loading out-of-tree module taints kernel. [ 56.275929] Key type ._llcrypt registered [ 56.277692] Key type .llcrypt registered [ 56.316860] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 63.424745] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 64.021575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.027437] alg: No test for adler32 (adler32-zlib) [ 64.977484] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 65.292772] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 66.895150] Key type lgssc registered [ 67.411739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.878269] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 74.735544] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 78.157572] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.673788] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.290405] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 88.919425] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 93.618679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.642627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 93.653564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 94.740488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 94.753865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 94.789400] Lustre: lustre-MDT0000: new disk, initializing [ 94.812733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 94.818560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 96.160193] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.398457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.425680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 101.455947] Lustre: 6485:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 101.468078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 101.470453] Lustre: Skipped 1 previous similar message [ 101.506740] Lustre: lustre-MDT0001: new disk, initializing [ 101.525613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 101.532840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 101.536485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 102.849119] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 105.006694] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 108.064480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.088206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 108.177393] Lustre: lustre-OST0000: new disk, initializing [ 108.179780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 108.197461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 109.999220] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 110.123811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 110.126145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 110.142684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 115.550978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.581738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 115.616696] Lustre: lustre-OST0001: new disk, initializing [ 115.618177] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 115.631692] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 116.525630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 116.529777] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 116.552542] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 117.456320] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 123.391093] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 130.094084] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 137.046300] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing check_logdir /tmp/testlogs/ [ 138.539253] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing yml_node [ 140.181337] Lustre: DEBUG MARKER: Client: 2.16.58.1 [ 140.991646] Lustre: DEBUG MARKER: MDS: 2.16.58.1 [ 141.827064] Lustre: DEBUG MARKER: OSS: 2.16.58.1 [ 142.374959] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 31 17:32:32 EDT 2025 [ 148.047133] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 148.558043] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 162.783865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 162.784245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 162.789625] Lustre: Skipped 2 previous similar messages [ 162.792445] Lustre: Skipped 1 previous similar message [ 166.268848] LustreError: 12186:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 166.297458] Lustre: server umount lustre-MDT0000 complete [ 167.535643] LustreError: 8392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756675976 with bad export cookie 4735599053668137807 [ 167.536655] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 167.540838] LustreError: 8392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 167.903940] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 167.904316] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 167.907963] Lustre: Skipped 2 previous similar messages [ 167.910035] Lustre: Skipped 2 previous similar messages [ 173.023560] LustreError: 7424:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 173.023718] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 173.031268] LustreError: 7424:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 173.036277] Lustre: Skipped 2 previous similar messages [ 173.688897] LustreError: 12388:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 173.690482] LustreError: 12388:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 173.755329] Lustre: server umount lustre-MDT0001 complete [ 175.059059] LustreError: 12589:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 175.060761] LustreError: 12589:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 175.077988] Lustre: server umount lustre-OST0000 complete [ 176.385660] Lustre: server umount lustre-OST0001 complete [ 178.498075] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 181.824135] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 185.448135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.833711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.000652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.214332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.680552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.710496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.801061] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 194.817230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 194.857250] Lustre: lustre-MDT0000: new disk, initializing [ 194.886923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.895133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 196.330485] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 200.494960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 200.517700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 200.549637] Lustre: 14969:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 200.645402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 201.936403] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 205.150633] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 206.452234] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 208.825453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.853329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 208.965702] Lustre: lustre-OST0000: new disk, initializing [ 208.967807] Lustre: Skipped 1 previous similar message [ 208.970481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 208.973789] Lustre: Skipped 2 previous similar messages [ 209.010860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.014041] Lustre: Skipped 1 previous similar message [ 210.493797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 210.497111] Lustre: Skipped 1 previous similar message [ 210.499811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 210.530943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 211.044723] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 214.182490] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 215.537140] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 215.616395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 216.935680] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 217.024382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 220.642685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 220.642802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 220.646723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 220.650655] Lustre: Skipped 1 previous similar message [ 223.725665] LustreError: 17908:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 223.727453] LustreError: 17908:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 223.792618] Lustre: server umount lustre-OST0000 complete [ 225.760796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 225.761490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 225.766825] Lustre: Skipped 2 previous similar messages [ 231.162991] LustreError: 18109:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 231.165316] LustreError: 18109:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 231.196519] Lustre: server umount lustre-MDT0000 complete [ 232.480334] LustreError: 14959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676041 with bad export cookie 4735599053668139326 [ 232.483376] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 232.484243] LustreError: 14959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 232.623148] Lustre: server umount lustre-MDT0001 complete [ 233.719088] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 17:34:03 (1756676043) [ 235.977063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.135128] LustreError: 18836:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 236.167267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.562635] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 240.352949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 241.738155] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 243.422868] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 244.500416] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 246.649607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 248.526731] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 250.186610] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 259.040087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 259.042996] LustreError: Skipped 1 previous similar message [ 259.045298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.051174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 259.053744] Lustre: Skipped 3 previous similar messages [ 261.092493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 262.510074] LustreError: 21080:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 262.511643] LustreError: 21080:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 262.567572] Lustre: server umount lustre-OST0000 complete [ 266.209276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 269.902460] Lustre: server umount lustre-MDT0000 complete [ 271.023020] LustreError: 18817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676080 with bad export cookie 4735599053668140201 [ 271.025403] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 271.028272] LustreError: 18817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 271.160700] Lustre: server umount lustre-MDT0001 complete [ 274.484653] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 275.495992] Key type lgssc unregistered [ 275.634376] LNet: 22015:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 275.636442] LNetError: 22015:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 275.646325] LNet: Removed LNI 192.168.203.157@tcp [ 275.970123] Key type .llcrypt unregistered [ 275.971802] Key type ._llcrypt unregistered [ 284.629923] Key type ._llcrypt registered [ 284.631351] Key type .llcrypt registered [ 284.667974] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 284.970713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 284.988664] alg: No test for adler32 (adler32-zlib) [ 285.836518] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 285.919579] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 287.495145] Key type lgssc registered [ 287.870096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 290.343271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 290.350286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 291.488270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 292.741540] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 295.436663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 295.564302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 296.784880] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 298.483644] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 299.705122] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 301.884177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 302.003117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 303.749821] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 305.416658] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 316.326832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 320.479741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 320.482155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.486546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 321.507225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 321.507663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 321.513027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.611082] LustreError: 26094:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 325.668178] Lustre: server umount lustre-OST0000 complete [ 331.744927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 331.744992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.745556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.745560] Lustre: Skipped 1 previous similar message [ 331.758756] Lustre: Skipped 1 previous similar message [ 332.989519] LustreError: 26296:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 332.992063] LustreError: 26296:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 333.022831] Lustre: server umount lustre-MDT0000 complete [ 334.329335] LustreError: 23366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676143 with bad export cookie 6025301522185593294 [ 334.333662] LustreError: 23366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 334.334565] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 334.394525] LustreError: 26499:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 334.397679] LustreError: 26499:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 334.483448] Lustre: server umount lustre-MDT0001 complete [ 338.511173] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 339.562666] Key type lgssc unregistered [ 339.712443] LNet: 27031:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.715190] LNetError: 27031:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 339.722299] LNet: Removed LNI 192.168.203.157@tcp [ 340.050130] Key type .llcrypt unregistered [ 340.051661] Key type ._llcrypt unregistered [ 349.396921] Key type ._llcrypt registered [ 349.398522] Key type .llcrypt registered [ 349.435201] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 349.797637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 349.836476] alg: No test for adler32 (adler32-zlib) [ 350.709124] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 350.805922] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 352.399123] Key type lgssc registered [ 352.778631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 355.390792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 355.399045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 356.549135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 357.792974] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 360.859547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 360.998261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 362.305666] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 364.161205] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 365.388370] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 369.976158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 370.097329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 372.030519] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 374.086633] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 382.379950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 386.528545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 386.531809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.537145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 387.553073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 387.554068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 387.556070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 388.269700] LustreError: 31017:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 388.327616] Lustre: server umount lustre-OST0000 complete [ 391.648026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 391.651306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 391.655366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 391.657381] Lustre: Skipped 1 previous similar message [ 391.688858] LustreError: 31218:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 391.691583] LustreError: 31218:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 391.722559] Lustre: server umount lustre-MDT0000 complete [ 392.672057] LustreError: 28405:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 392.679149] LustreError: 28405:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 393.033405] LustreError: 31219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676202 with bad export cookie 12242908009011115509 [ 393.037041] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 393.037554] LustreError: 31219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 393.103255] LustreError: 31420:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 393.105870] LustreError: 31420:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 393.196708] Lustre: server umount lustre-MDT0001 complete [ 397.287171] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 398.466976] Key type lgssc unregistered [ 398.607497] LNet: 31950:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.610797] LNetError: 31950:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 398.620578] LNet: Removed LNI 192.168.203.157@tcp [ 398.954151] Key type .llcrypt unregistered [ 398.955629] Key type ._llcrypt unregistered [ 408.251500] Key type ._llcrypt registered [ 408.252867] Key type .llcrypt registered [ 408.293445] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 408.684735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 408.716574] alg: No test for adler32 (adler32-zlib) [ 409.577964] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 409.664620] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 411.247169] Key type lgssc registered [ 411.648919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 414.491229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 414.500483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.698816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 417.193030] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 420.642640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 420.806624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 422.306830] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 424.290609] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 425.655757] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 428.214249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 428.347955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.393572] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 432.164939] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 440.677049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 445.922044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.922661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 445.929053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 445.934172] Lustre: Skipped 1 previous similar message [ 447.406453] LustreError: 35589:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 447.470057] Lustre: server umount lustre-OST0000 complete [ 450.812696] LustreError: 35789:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 450.815573] LustreError: 35789:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 450.838593] Lustre: server umount lustre-MDT0000 complete [ 451.040494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.041332] LustreError: 33319:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.045525] Lustre: Skipped 2 previous similar messages [ 451.052808] LustreError: 33319:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 452.064141] LustreError: 33303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676261 with bad export cookie 13440660345953957896 [ 452.067135] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 452.068558] LustreError: 33303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 452.125535] LustreError: 35990:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 452.127508] LustreError: 35990:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 452.200901] Lustre: server umount lustre-MDT0001 complete [ 455.762141] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 456.809525] Key type lgssc unregistered [ 456.943455] LNet: 36522:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 456.947167] LNetError: 36522:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 456.958272] LNet: Removed LNI 192.168.203.157@tcp [ 457.250309] Key type .llcrypt unregistered [ 457.252087] Key type ._llcrypt unregistered [ 466.035076] Key type ._llcrypt registered [ 466.036142] Key type .llcrypt registered [ 466.073176] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 466.406169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 466.449857] alg: No test for adler32 (adler32-zlib) [ 467.307794] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 467.384671] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 468.967165] Key type lgssc registered [ 469.342651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 471.986317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 471.992733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.141410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.501634] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 477.396153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 477.545250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 478.921836] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 480.695490] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 481.886679] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 484.140667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 484.250241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 486.083781] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 487.839180] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 499.627090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 507.872237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 507.875537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.880765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 508.807241] Lustre: lustre-OST0000: Not available for connect from 192.168.203.57@tcp (stopping) [ 509.921687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 509.921938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.929070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 511.982101] LustreError: 40158:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 512.041642] Lustre: server umount lustre-OST0000 complete [ 524.835779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.837068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 524.840084] Lustre: Skipped 1 previous similar message [ 524.841951] Lustre: Skipped 1 previous similar message [ 529.887886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 529.890603] Lustre: Skipped 2 previous similar messages [ 531.004940] LustreError: 40535:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 531.006896] LustreError: 40535:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 531.031309] Lustre: server umount lustre-MDT0000 complete [ 532.267233] LustreError: 37873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676341 with bad export cookie 9555539484620834081 [ 532.270214] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 532.272160] LustreError: 37873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 538.362829] LustreError: 40736:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 538.365803] LustreError: 40736:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 538.442193] Lustre: server umount lustre-MDT0001 complete [ 541.923634] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 543.009243] Key type lgssc unregistered [ 543.146330] LNet: 41269:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 543.148640] LNetError: 41269:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 543.155322] LNet: Removed LNI 192.168.203.157@tcp [ 543.446214] Key type .llcrypt unregistered [ 543.447160] Key type ._llcrypt unregistered [ 552.163526] Key type ._llcrypt registered [ 552.164968] Key type .llcrypt registered [ 552.201611] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 552.539677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 552.588120] alg: No test for adler32 (adler32-zlib) [ 553.448495] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 553.532829] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 555.119225] Key type lgssc registered [ 555.516345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 558.313366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 558.321572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.492982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 560.932302] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 563.991970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 564.128332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 565.538622] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 567.408421] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 568.697236] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 571.076981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 571.203468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.254731] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 575.236494] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 584.554580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 589.795117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 589.796083] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.800083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 589.804711] Lustre: Skipped 1 previous similar message [ 589.808938] Lustre: Skipped 2 previous similar messages [ 590.727667] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.57@tcp (stopping) [ 594.172222] LustreError: 44903:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 594.205601] Lustre: server umount lustre-MDT0000 complete [ 594.913749] LustreError: 42635:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.920502] LustreError: 42635:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 595.596660] LustreError: 42618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676404 with bad export cookie 8464692726067445244 [ 595.599969] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.602517] LustreError: 42618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 595.838475] LustreError: 42634:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.847995] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.57@tcp (stopping) [ 600.032055] LustreError: 42635:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.032466] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.045752] Lustre: Skipped 1 previous similar message [ 600.048365] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 601.732042] LustreError: 45106:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 601.735136] LustreError: 45106:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 601.817898] Lustre: server umount lustre-MDT0001 complete [ 619.756370] LustreError: 45329:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 619.758838] LustreError: 45329:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 619.832236] Lustre: server umount lustre-OST0000 complete [ 625.729387] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 626.834155] Key type lgssc unregistered [ 626.975494] LNet: 46166:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.978675] LNetError: 46166:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 626.990461] LNet: Removed LNI 192.168.203.157@tcp [ 627.313819] Key type .llcrypt unregistered [ 627.315289] Key type ._llcrypt unregistered [ 637.398942] Key type ._llcrypt registered [ 637.400740] Key type .llcrypt registered [ 637.446563] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 637.884188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 637.896918] alg: No test for adler32 (adler32-zlib) [ 638.762195] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 638.852720] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 640.447277] Key type lgssc registered [ 640.869945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 643.741983] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 643.749683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 660.191288] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 666.335663] LustreError: 47518:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 696.031269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 698.105269] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 700.016239] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 863.412483] LustreError: 48227:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 863.469826] Lustre: server umount lustre-OST0000 complete [ 868.561519] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 869.585888] Key type lgssc unregistered [ 869.720324] LNet: 49063:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.722687] LNetError: 49063:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.731267] LNet: Removed LNI 192.168.203.157@tcp [ 870.011656] Key type .llcrypt unregistered [ 870.012769] Key type ._llcrypt unregistered [ 878.766272] Key type ._llcrypt registered [ 878.767768] Key type .llcrypt registered [ 878.808038] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 879.189500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 879.204635] alg: No test for adler32 (adler32-zlib) [ 880.063819] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 880.155887] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 881.735135] Key type lgssc registered [ 882.112985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 884.805662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 884.812944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 885.978483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 887.375694] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 890.326287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 890.461205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 891.852646] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 893.597339] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 894.788322] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 896.986443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 897.106873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 898.900451] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 900.623302] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 905.697135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 905.697569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 905.705703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 907.437227] LustreError: 52678:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 907.492284] Lustre: server umount lustre-OST0000 complete [ 908.695869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.696660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 908.700056] Lustre: Skipped 1 previous similar message [ 913.887964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 913.890131] Lustre: Skipped 2 previous similar messages [ 914.812572] LustreError: 52880:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 914.815382] LustreError: 52880:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 914.842644] Lustre: server umount lustre-MDT0000 complete [ 916.107645] LustreError: 52052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676725 with bad export cookie 12904500559050989483 [ 916.110975] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 916.112985] LustreError: 52052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 916.175677] LustreError: 53081:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 916.177768] LustreError: 53081:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 916.260362] Lustre: server umount lustre-MDT0001 complete [ 920.066993] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 921.209329] Key type lgssc unregistered [ 921.348571] LNet: 53612:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.351508] LNetError: 53612:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.360330] LNet: Removed LNI 192.168.203.157@tcp [ 921.661687] Key type .llcrypt unregistered [ 921.662966] Key type ._llcrypt unregistered [ 930.082663] Key type ._llcrypt registered [ 930.084130] Key type .llcrypt registered [ 930.121529] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 930.428195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 930.448942] alg: No test for adler32 (adler32-zlib) [ 931.313793] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 931.404937] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 932.983158] Key type lgssc registered [ 933.347714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 935.929774] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 935.936187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 953.439277] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 959.583179] LustreError: 54954:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 989.279140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.015509] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 992.632749] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 994.681500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 994.827431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 994.850355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 996.038893] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 998.771662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 998.888194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1000.190629] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1001.799261] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1002.870177] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1004.000154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1004.002891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.007441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1005.023939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1005.026155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.029595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1005.549893] LustreError: 57227:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1005.607917] Lustre: server umount lustre-OST0000 complete [ 1019.435098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.438729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1023.520427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.520696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1023.526008] Lustre: Skipped 1 previous similar message [ 1025.599379] LustreError: 57609:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1025.602340] LustreError: 57609:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1025.631608] Lustre: server umount lustre-MDT0000 complete [ 1026.839111] LustreError: 57229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676836 with bad export cookie 5008011535695745324 [ 1026.842903] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1026.843913] LustreError: 57229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1041.375197] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1041.410886] LustreError: 57810:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1041.412961] LustreError: 57810:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1041.503671] Lustre: server umount lustre-MDT0001 complete [ 1045.331321] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1046.400188] Key type lgssc unregistered [ 1046.532409] LNet: 58343:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.535223] LNetError: 58343:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.545331] LNet: Removed LNI 192.168.203.157@tcp [ 1046.826473] Key type .llcrypt unregistered [ 1046.827493] Key type ._llcrypt unregistered [ 1055.752113] Key type ._llcrypt registered [ 1055.753181] Key type .llcrypt registered [ 1055.792056] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1056.108769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1056.129161] alg: No test for adler32 (adler32-zlib) [ 1057.000226] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1057.101369] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1058.687150] Key type lgssc registered [ 1059.057360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1061.642928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1061.648534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1062.800186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1064.149677] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1066.980632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1067.116292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1068.468718] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1070.337449] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1071.479908] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1073.781056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1073.900979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.867806] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1077.724744] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1088.230425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1092.576317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1092.580619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.585355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1093.601653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1093.602803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1093.603032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.611612] Lustre: Skipped 1 previous similar message [ 1096.300893] LustreError: 61981:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1096.362928] Lustre: server umount lustre-OST0000 complete [ 1097.695890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1097.698941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.704569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1103.745859] LustreError: 62183:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1103.748919] LustreError: 62183:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1103.778039] Lustre: server umount lustre-MDT0000 complete [ 1103.841236] LustreError: 59712:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1103.849811] LustreError: 59712:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1105.063608] LustreError: 59694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676914 with bad export cookie 15501342637730444724 [ 1105.068443] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1105.146720] LustreError: 62384:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1105.148669] LustreError: 62384:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1105.232267] Lustre: server umount lustre-MDT0001 complete [ 1108.723351] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1109.789323] Key type lgssc unregistered [ 1109.927320] LNet: 62915:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.929354] LNetError: 62915:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.935275] LNet: Removed LNI 192.168.203.157@tcp [ 1110.213049] Key type .llcrypt unregistered [ 1110.214192] Key type ._llcrypt unregistered [ 1122.177955] Key type ._llcrypt registered [ 1122.178995] Key type .llcrypt registered [ 1122.220396] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1122.742231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1122.767455] alg: No test for adler32 (adler32-zlib) [ 1123.622717] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1123.717489] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1125.303170] Key type lgssc registered [ 1125.702262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1128.278449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1128.284980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.435529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1130.656243] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1133.384565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1133.507933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1134.794688] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1136.630694] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1137.844967] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1140.082013] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1140.195934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.022480] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1143.837721] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1154.533991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1161.657268] Lustre: Failing over lustre-MDT0000 [ 1161.778157] LustreError: 66841:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1161.807479] Lustre: server umount lustre-MDT0000 complete [ 1164.255888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1164.258129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.262019] LustreError: 64573:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.266221] LustreError: 64573:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1164.769633] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.770201] LustreError: 64569:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.773085] Lustre: Skipped 1 previous similar message [ 1164.780127] LustreError: 64569:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1164.818014] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.857633] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1164.908860] Lustre: *** cfs_fail_loc=135, val=0*** [ 1164.949954] LustreError: 67351:0:(obd_config.c:782:class_setup()) setup lustre-MDT0000 failed (-2) [ 1164.953374] LustreError: 67351:0:(obd_config.c:1996:class_config_llog_handler()) MGC192.168.203.157@tcp: cfg command failed: rc = -2 [ 1164.958096] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1164.958096] [ 1164.961260] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1164.964907] LustreError: 67339:0:(tgt_mount.c:1710:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1164.968200] LustreError: 67339:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -2 [ 1164.970737] LustreError: 67339:0:(obd_config.c:835:class_cleanup()) Device 5 not setup [ 1164.979306] Lustre: server umount lustre-MDT0000 complete [ 1164.980491] LustreError: 67339:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1167.118195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1167.158590] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1167.258535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1167.291594] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1168.556587] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1169.789410] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1172.452812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1172.457970] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1172.474371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1173.983840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1173.986562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1173.990150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1177.570087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1177.570179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1177.570618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1177.577898] Lustre: Skipped 1 previous similar message [ 1179.438106] LustreError: 68159:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1179.441274] LustreError: 68159:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1179.499126] Lustre: server umount lustre-OST0000 complete [ 1182.689079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1182.689183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.689734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1182.696063] Lustre: Skipped 1 previous similar message [ 1187.808066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1187.810120] Lustre: Skipped 2 previous similar messages [ 1192.928119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1192.930499] Lustre: Skipped 1 previous similar message [ 1194.975131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1195.007737] LustreError: 68360:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1195.010465] LustreError: 68360:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1195.037884] Lustre: server umount lustre-MDT0000 complete [ 1196.324115] LustreError: 64555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677005 with bad export cookie 14935889065086266868 [ 1196.327459] LustreError: 64555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1196.328180] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1196.480909] Lustre: server umount lustre-MDT0001 complete [ 1199.978973] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1201.090435] Key type lgssc unregistered [ 1201.218440] LNet: 69094:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1201.221905] LNetError: 69094:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.229406] LNet: Removed LNI 192.168.203.157@tcp [ 1201.536045] Key type .llcrypt unregistered [ 1201.537075] Key type ._llcrypt unregistered [ 1210.109687] Key type ._llcrypt registered [ 1210.111072] Key type .llcrypt registered [ 1210.144821] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1210.567909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1210.586516] alg: No test for adler32 (adler32-zlib) [ 1211.440041] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1211.526107] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1213.103134] Key type lgssc registered [ 1213.454192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1215.962081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1215.968064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.111771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.476703] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1221.228188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1221.349135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1222.725687] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1224.539482] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1225.791803] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1227.927186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1228.038616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1229.778995] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1231.415853] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1243.432920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1249.247584] Lustre: Failing over lustre-MDT0000 [ 1249.380288] LustreError: 72734:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1249.408889] Lustre: server umount lustre-MDT0000 complete [ 1252.226723] LustreError: 72569:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.232468] LustreError: 72569:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1252.319700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1252.322267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.375120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1252.413535] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1252.466326] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1252.528360] LustreError: 73242:0:(obd_config.c:782:class_setup()) setup lustre-MDT0000 failed (-9) [ 1252.530452] LustreError: 73242:0:(obd_config.c:1996:class_config_llog_handler()) MGC192.168.203.157@tcp: cfg command failed: rc = -9 [ 1252.533405] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1252.533405] [ 1252.536606] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1252.541837] LustreError: 73230:0:(tgt_mount.c:1710:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1252.544203] LustreError: 73230:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -9 [ 1252.547197] LustreError: 73230:0:(obd_config.c:835:class_cleanup()) Device 5 not setup [ 1252.555085] Lustre: server umount lustre-MDT0000 complete [ 1252.556223] LustreError: 73230:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1253.856871] LustreError: 71840:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.857571] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.861319] LustreError: 71840:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1253.866457] Lustre: Skipped 1 previous similar message [ 1254.642227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1254.682629] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1254.775209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1254.798712] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1256.108533] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1257.340965] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1260.005856] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1260.013867] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1260.029333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1262.048071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1262.051792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.057751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1265.124421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1265.128884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1265.131140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.051641] LustreError: 74051:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1267.054421] LustreError: 74051:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1267.110381] Lustre: server umount lustre-OST0000 complete [ 1270.241082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.241459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1270.241673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1270.241677] Lustre: Skipped 1 previous similar message [ 1270.244359] Lustre: Skipped 1 previous similar message [ 1274.427354] LustreError: 74252:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1274.429229] LustreError: 74252:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1274.452566] Lustre: server umount lustre-MDT0000 complete [ 1275.360026] LustreError: 71840:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.368077] LustreError: 71840:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1275.641410] LustreError: 70446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677084 with bad export cookie 12576382762043221685 [ 1275.644488] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1275.645739] LustreError: 70446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1275.778093] Lustre: server umount lustre-MDT0001 complete [ 1279.353224] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1280.373961] Key type lgssc unregistered [ 1280.508359] LNet: 74986:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.510543] LNetError: 74986:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.517293] LNet: Removed LNI 192.168.203.157@tcp [ 1280.812781] Key type .llcrypt unregistered [ 1280.813907] Key type ._llcrypt unregistered [ 1284.965657] Key type ._llcrypt registered [ 1284.967139] Key type .llcrypt registered [ 1285.003288] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1285.096386] Key type .llcrypt unregistered [ 1285.097440] Key type ._llcrypt unregistered [ 1293.630769] Key type ._llcrypt registered [ 1293.631808] Key type .llcrypt registered [ 1293.671357] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1294.034971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1294.063519] alg: No test for adler32 (adler32-zlib) [ 1294.912268] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1294.984450] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1296.559100] Key type lgssc registered [ 1296.881771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1299.386205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1299.392822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1300.569285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1301.908684] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1304.714848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1304.860523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1306.168220] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1307.850231] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1308.980632] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1311.206892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1311.323750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1313.028940] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1314.710308] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1325.672776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1336.293093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.293432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1336.294475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1336.298619] Lustre: Skipped 1 previous similar message [ 1336.304649] LustreError: Skipped 1 previous similar message [ 1339.245489] LustreError: 79092:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1339.320586] Lustre: server umount lustre-OST0000 complete [ 1340.895819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1340.898136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.901705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1340.903343] Lustre: Skipped 1 previous similar message [ 1346.528174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.530154] Lustre: Skipped 3 previous similar messages [ 1346.617994] LustreError: 79294:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1346.620564] LustreError: 79294:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1346.646274] Lustre: server umount lustre-MDT0000 complete [ 1347.877111] LustreError: 76804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677157 with bad export cookie 13009397685066039953 [ 1347.879348] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1347.882746] LustreError: 76804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1347.943709] LustreError: 79497:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1347.945532] LustreError: 79497:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1348.017598] Lustre: server umount lustre-MDT0001 complete [ 1351.650396] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1352.690151] Key type lgssc unregistered [ 1352.816403] LNet: 80029:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1352.819278] LNetError: 80029:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1352.828363] LNet: Removed LNI 192.168.203.157@tcp [ 1353.142300] Key type .llcrypt unregistered [ 1353.143642] Key type ._llcrypt unregistered [ 1361.988679] Key type ._llcrypt registered [ 1361.989780] Key type .llcrypt registered [ 1362.022368] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1362.344762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1362.378546] alg: No test for adler32 (adler32-zlib) [ 1363.225356] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1363.304570] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1364.887139] Key type lgssc registered [ 1365.425727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1368.829907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1368.837065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1369.995856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.339725] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1374.344779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1374.462977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1375.697190] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1377.429637] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1378.661765] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1381.312704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1381.475044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1383.718667] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1385.546940] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1395.813333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1400.289319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1400.291667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.295540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1401.316477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1401.317931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1401.318579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.332122] Lustre: Skipped 1 previous similar message [ 1404.397961] LustreError: 83665:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1404.478851] Lustre: server umount lustre-OST0000 complete [ 1406.432693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1406.433574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1406.435960] Lustre: Skipped 1 previous similar message [ 1406.441422] Lustre: Skipped 1 previous similar message [ 1411.552295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1411.555137] Lustre: Skipped 1 previous similar message [ 1416.672119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1416.673926] Lustre: Skipped 1 previous similar message [ 1420.255130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1420.293825] LustreError: 83868:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1420.295993] LustreError: 83868:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1420.325798] Lustre: server umount lustre-MDT0000 complete [ 1421.588522] LustreError: 81379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677230 with bad export cookie 17236396130764177691 [ 1421.592405] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1421.592967] LustreError: 81379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1421.670052] LustreError: 84070:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1421.672382] LustreError: 84070:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1421.759375] Lustre: server umount lustre-MDT0001 complete [ 1425.441181] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1426.620791] Key type lgssc unregistered [ 1426.769503] LNet: 84601:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.772310] LNetError: 84601:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1426.782363] LNet: Removed LNI 192.168.203.157@tcp [ 1427.126281] Key type .llcrypt unregistered [ 1427.127430] Key type ._llcrypt unregistered [ 1436.266765] Key type ._llcrypt registered [ 1436.267759] Key type .llcrypt registered [ 1436.303678] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1436.610730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1436.618954] alg: No test for adler32 (adler32-zlib) [ 1437.464849] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1437.541157] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1439.119144] Key type lgssc registered [ 1439.482033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1442.048089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1442.054277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.210820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1444.494244] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1447.412538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1447.537212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1448.838918] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1450.612882] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1451.834608] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1454.142513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1454.255121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.080372] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1457.762587] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1469.606469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1478.112734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1478.115230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.121278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1480.161770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1480.163128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.164290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1482.795882] LustreError: 88239:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1482.860939] Lustre: server umount lustre-OST0000 complete [ 1485.281384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1485.282587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1485.291231] Lustre: Skipped 1 previous similar message [ 1485.295956] Lustre: Skipped 1 previous similar message [ 1490.169575] LustreError: 88442:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1490.171314] LustreError: 88442:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1490.196756] Lustre: server umount lustre-MDT0000 complete [ 1490.400285] LustreError: 85971:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.405340] LustreError: 85971:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1491.442742] LustreError: 87589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677300 with bad export cookie 17478739906384083255 [ 1491.447311] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1491.448279] LustreError: 87589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1491.515443] LustreError: 88644:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1491.517190] LustreError: 88644:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1491.612312] Lustre: server umount lustre-MDT0001 complete [ 1495.490928] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1496.529929] Key type lgssc unregistered [ 1496.663373] LNet: 89175:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1496.666331] LNetError: 89175:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1496.673265] LNet: Removed LNI 192.168.203.157@tcp [ 1496.969871] Key type .llcrypt unregistered [ 1496.970917] Key type ._llcrypt unregistered [ 1505.509952] Key type ._llcrypt registered [ 1505.510962] Key type .llcrypt registered [ 1505.543439] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1505.952923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1505.966049] alg: No test for adler32 (adler32-zlib) [ 1506.806133] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1506.888868] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1508.463157] Key type lgssc registered [ 1508.814813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1511.305097] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1511.312165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1528.799278] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1534.943246] LustreError: 90519:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 1564.639191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.562344] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1568.331105] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1570.151575] LustreError: 91376:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1570.203829] Lustre: server umount lustre-OST0000 complete [ 1572.938693] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 17:56:22 (1756677382) [ 1575.311994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1575.485252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1576.794987] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1579.755064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1579.884386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1581.184691] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1583.198554] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1584.650087] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1586.991395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1587.104360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1589.053053] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1590.985355] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1600.434388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1605.603422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1605.603582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1605.604345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1605.611689] Lustre: Skipped 1 previous similar message [ 1606.703336] LustreError: 94437:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1606.708311] LustreError: 94437:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1606.794898] Lustre: server umount lustre-OST0000 complete [ 1610.723057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1610.723184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1610.725429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1610.725440] Lustre: Skipped 1 previous similar message [ 1610.725854] LustreError: Skipped 1 previous similar message [ 1610.729850] Lustre: Skipped 1 previous similar message [ 1614.161537] LustreError: 94638:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1614.165530] LustreError: 94638:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1614.205371] Lustre: server umount lustre-MDT0000 complete [ 1615.842311] LustreError: 93313:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1615.852472] LustreError: 93313:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1616.049437] LustreError: 92029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677425 with bad export cookie 5586844652646033165 [ 1616.055274] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1616.056712] LustreError: 92029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1616.163126] LustreError: 94840:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 1616.165796] LustreError: 94840:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1616.258839] Lustre: server umount lustre-MDT0001 complete [ 1619.933072] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1620.972217] Key type lgssc unregistered [ 1621.106366] LNet: 95371:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.108618] LNetError: 95371:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1621.116302] LNet: Removed LNI 192.168.203.157@tcp [ 1621.436714] Key type .llcrypt unregistered [ 1621.437872] Key type ._llcrypt unregistered [ 1630.530549] Key type ._llcrypt registered [ 1630.532531] Key type .llcrypt registered [ 1630.570607] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1630.884586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1630.894744] alg: No test for adler32 (adler32-zlib) [ 1631.747842] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1631.846223] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1633.431111] Key type lgssc registered [ 1633.792885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1636.477178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1636.484159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1637.648824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1639.157060] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1642.185553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1642.313421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1643.679950] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1645.502323] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1646.784425] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1649.090365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1649.204902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1651.090869] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1652.937688] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1662.499761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1667.553601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1667.554755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.556559] LustreError: Skipped 1 previous similar message [ 1667.557430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1667.560945] Lustre: Skipped 1 previous similar message [ 1667.566100] Lustre: Skipped 1 previous similar message [ 1672.043110] LustreError: 99009:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1672.113684] Lustre: server umount lustre-OST0000 complete [ 1677.793306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1677.794219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1677.796571] Lustre: Skipped 1 previous similar message [ 1679.486392] LustreError: 99212:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1679.488392] LustreError: 99212:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1679.513785] Lustre: server umount lustre-MDT0000 complete [ 1680.745294] LustreError: 96723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677490 with bad export cookie 10282689708999812867 [ 1680.748669] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1680.751621] LustreError: 96723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1680.817681] LustreError: 99413:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1680.819515] LustreError: 99413:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1680.898169] Lustre: server umount lustre-MDT0001 complete [ 1684.631346] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1685.776851] Key type lgssc unregistered [ 1685.912366] LNet: 99945:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.914630] LNetError: 99945:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.924315] LNet: Removed LNI 192.168.203.157@tcp [ 1686.216162] Key type .llcrypt unregistered [ 1686.217854] Key type ._llcrypt unregistered [ 1693.569975] Key type ._llcrypt registered [ 1693.571496] Key type .llcrypt registered [ 1693.613642] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1693.989562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1694.022783] alg: No test for adler32 (adler32-zlib) [ 1694.880959] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1694.965202] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1696.543164] Key type lgssc registered [ 1696.930507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1699.661799] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1699.668630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1717.151260] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1723.295207] LustreError: 101145:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 1754.015139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.772849] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1757.522713] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1759.936289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.030933] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1760.035022] LustreError: 102056:0:(tgt_mount.c:1710:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1760.037511] LustreError: 102056:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -2 [ 1760.039870] LustreError: 102056:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-MDT0000 [ 1760.096849] Lustre: server umount lustre-MDT0000 complete [ 1760.098231] LustreError: 102056:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1773.023494] LustreError: 102489:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1773.088814] Lustre: server umount lustre-OST0000 complete [ 1775.804704] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 1778.599413] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1782.043532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1784.297957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1786.298477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1788.356206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1790.598282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1790.624222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.706092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1790.716367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1790.756690] Lustre: lustre-MDT0000: new disk, initializing [ 1790.784813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.793149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1792.027333] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1796.020094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1796.043371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1796.075962] Lustre: 104821:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1796.096867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1796.099150] Lustre: Skipped 1 previous similar message [ 1796.135555] Lustre: lustre-MDT0001: new disk, initializing [ 1796.167515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1796.180601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1796.183884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1797.516279] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1800.581074] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1801.831989] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1804.171489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1804.200344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1804.315675] Lustre: lustre-OST0000: new disk, initializing [ 1804.318951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1804.350739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1806.347823] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1809.495224] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1809.913124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1809.916295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1809.949138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1810.841819] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1810.922212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1812.201307] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1812.272446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1815.009721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1815.010703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.012039] LustreError: Skipped 1 previous similar message [ 1815.017777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1815.021577] Lustre: Skipped 1 previous similar message [ 1818.926153] LustreError: 107761:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1818.928017] LustreError: 107761:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1818.992930] Lustre: server umount lustre-OST0000 complete [ 1821.664068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1821.667257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.670860] Lustre: Skipped 1 previous similar message [ 1821.672541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1825.248703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1825.249069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1826.431804] LustreError: 107963:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1826.434080] LustreError: 107963:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1826.464255] Lustre: server umount lustre-MDT0000 complete [ 1827.721442] LustreError: 104811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677637 with bad export cookie 6766277361082929552 [ 1827.725319] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.725666] LustreError: 104811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1827.889891] Lustre: server umount lustre-MDT0001 complete [ 1830.702516] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:00:40 (1756677640) [ 1831.760289] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1837.414806] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 1841.493544] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1845.751077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1848.280188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1850.304040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1852.387318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1854.650572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1854.676349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.760548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1854.771827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1854.808977] Lustre: lustre-MDT0000: new disk, initializing [ 1854.831534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.838116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1856.135316] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1860.219331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1860.246543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1860.279836] Lustre: 111306:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1860.302611] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1860.304413] Lustre: Skipped 1 previous similar message [ 1860.340464] Lustre: lustre-MDT0001: new disk, initializing [ 1860.380295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1860.384096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1861.638496] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1864.758813] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1866.012142] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1868.216194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1868.241247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1868.338909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1868.362712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1868.365074] Lustre: Skipped 1 previous similar message [ 1870.266622] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1871.737689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1871.742249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1871.776357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1873.429656] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1874.781276] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1874.857575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1876.163641] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1876.234681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1876.961274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1876.961351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.964516] LustreError: Skipped 1 previous similar message [ 1876.970131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1876.974435] Lustre: Skipped 2 previous similar messages [ 1882.083214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1882.086530] Lustre: Skipped 1 previous similar message [ 1891.295137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1891.312088] LustreError: 114241:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1891.314520] LustreError: 114241:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1891.378204] Lustre: server umount lustre-OST0000 complete [ 1896.415841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1896.417957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1896.421879] Lustre: Skipped 1 previous similar message [ 1896.423545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1896.425521] Lustre: Skipped 2 previous similar messages [ 1898.822711] LustreError: 114443:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1898.827644] LustreError: 114443:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1898.862580] Lustre: server umount lustre-MDT0000 complete [ 1900.132128] LustreError: 111296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677709 with bad export cookie 6766277361082930420 [ 1900.134708] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1900.135796] LustreError: 111296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1900.276972] Lustre: server umount lustre-MDT0001 complete [ 1903.550599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1903.745508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1905.210266] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1908.488341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1909.924445] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1911.674559] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1912.881947] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1915.522258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1917.850154] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1919.769203] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1932.553601] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1934.307599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.308260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1934.310798] Lustre: Skipped 2 previous similar messages [ 1934.312370] Lustre: Skipped 2 previous similar messages [ 1934.314035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1935.597585] LustreError: 117481:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1935.599653] LustreError: 117481:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1935.676958] Lustre: server umount lustre-OST0000 complete [ 1939.029548] Lustre: server umount lustre-MDT0000 complete [ 1939.424524] LustreError: 115119:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1939.431875] LustreError: 115119:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1940.277574] LustreError: 115143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677749 with bad export cookie 6766277361082931288 [ 1940.282619] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1940.431620] Lustre: server umount lustre-MDT0001 complete [ 1944.240728] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1945.369995] Key type lgssc unregistered [ 1945.512349] LNet: 118415:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.514876] LNetError: 118415:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.526439] LNet: Removed LNI 192.168.203.157@tcp [ 1945.863146] Key type .llcrypt unregistered [ 1945.864239] Key type ._llcrypt unregistered [ 1951.743823] Key type ._llcrypt registered [ 1951.744970] Key type .llcrypt registered [ 1951.782189] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 1958.348113] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1958.678248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1958.702078] alg: No test for adler32 (adler32-zlib) [ 1959.552380] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1959.632401] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1961.215221] Key type lgssc registered [ 1961.613768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1964.575880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1966.700452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1968.647945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1970.713403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1972.907070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1972.924910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1972.930330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1974.015714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1974.028608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1974.070039] Lustre: lustre-MDT0000: new disk, initializing [ 1974.096396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1974.105404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1975.361682] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1979.362073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1979.387782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1979.421041] Lustre: 121514:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1979.442607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1979.444284] Lustre: Skipped 1 previous similar message [ 1979.477652] Lustre: lustre-MDT0001: new disk, initializing [ 1979.509782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1979.521765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1979.524978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1980.776252] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1983.980222] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1985.464526] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1988.044958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1988.082888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1988.207951] Lustre: lustre-OST0000: new disk, initializing [ 1988.210509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1988.241503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.302952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1989.305671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1989.339285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1990.281318] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1993.718518] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1995.152510] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1995.235876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1996.616373] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1996.705155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1999.533814] LustreError: 124455:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1999.602197] Lustre: server umount lustre-OST0000 complete [ 2004.960703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.961503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2004.964661] Lustre: Skipped 1 previous similar message [ 2004.968732] Lustre: Skipped 1 previous similar message [ 2010.080230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.082135] Lustre: Skipped 1 previous similar message [ 2015.199107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2015.200083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.202867] Lustre: Skipped 1 previous similar message [ 2015.232599] LustreError: 124655:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2015.234355] LustreError: 124655:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2015.269781] Lustre: server umount lustre-MDT0000 complete [ 2016.223111] Lustre: 119732:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756677809/real 1756677809] req@00000000080bfe2c x1842010147284864/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1756677825 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2016.229276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.504160] LustreError: 122296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677825 with bad export cookie 7525060209484042454 [ 2016.507214] LustreError: 122296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2016.507449] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2016.569837] LustreError: 124859:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2016.571729] LustreError: 124859:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2016.649961] Lustre: server umount lustre-MDT0001 complete [ 2019.347869] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:03:49 (1756677829) [ 2021.541389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.699209] LustreError: 125529:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.704368] LustreError: 125529:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 2021.720394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2023.071806] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2025.905669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2026.055843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2027.594656] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2029.497941] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2030.704934] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2036.192753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.192868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2036.193440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.196649] Lustre: Skipped 1 previous similar message [ 2037.628921] LustreError: 126870:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2037.630756] LustreError: 126870:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2037.655542] Lustre: server umount lustre-MDT0000 complete [ 2038.932331] LustreError: 125553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677848 with bad export cookie 7525060209484043322 [ 2038.936509] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2038.936810] LustreError: 125553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2039.070809] Lustre: server umount lustre-MDT0001 complete [ 2041.911435] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:04:11 (1756677851) [ 2044.278639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2061.791564] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2067.935369] LustreError: 127717:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 2096.607290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.457736] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2100.144544] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2100.748062] LustreError: 128381:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2100.749900] LustreError: 128381:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2100.804116] Lustre: server umount lustre-OST0000 complete [ 2103.408528] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:05:13 (1756677913) [ 2105.658246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.813088] LustreError: 129052:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.206837] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2110.015832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2111.390687] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2113.127382] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2114.304829] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2116.437686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2116.545273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.547185] Lustre: Skipped 2 previous similar messages [ 2118.262771] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2120.061495] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2125.794913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2125.796142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2125.799266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2125.799277] Lustre: Skipped 3 previous similar messages [ 2125.801608] Lustre: Skipped 2 previous similar messages [ 2128.640683] LustreError: 131299:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2128.642727] LustreError: 131299:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2128.670985] Lustre: server umount lustre-MDT0000 complete [ 2130.080338] LustreError: 131213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677939 with bad export cookie 7525060209484043973 [ 2130.083950] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2130.086410] LustreError: 131213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2130.249229] Lustre: server umount lustre-MDT0001 complete [ 2141.756349] Lustre: server umount lustre-OST0000 complete [ 2144.399047] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:05:54 (1756677954) [ 2146.481152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.617085] LustreError: 132372:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.884911] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2150.667265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2150.791117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2150.793100] Lustre: Skipped 1 previous similar message [ 2152.152642] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2153.931920] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2155.142908] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2157.308057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2159.136443] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2160.865574] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2162.221359] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2170.727017] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2171.503618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2172.720198] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2172.787012] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2175.970445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2175.970466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2175.975767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2175.978911] Lustre: Skipped 3 previous similar messages [ 2179.433474] LustreError: 135176:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2179.435290] LustreError: 135176:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2179.499708] Lustre: server umount lustre-OST0000 complete [ 2186.837119] Lustre: server umount lustre-MDT0000 complete [ 2188.081167] LustreError: 132353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677997 with bad export cookie 7525060209484044988 [ 2188.084292] LustreError: 132353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2188.084475] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2188.222533] Lustre: server umount lustre-MDT0001 complete [ 2190.844120] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:06:40 (1756678000) [ 2193.035490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2210.527405] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2216.671380] LustreError: 136222:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 2245.343285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.345176] Lustre: Skipped 1 previous similar message [ 2247.040677] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2248.759775] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2250.912860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2251.067231] LustreError: 136249:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.113134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2252.292111] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2254.937872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2256.202104] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2257.880344] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2259.030823] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2260.347937] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2260.429204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2261.601925] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2261.663725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2264.545500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.546873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2264.550798] Lustre: Skipped 4 previous similar messages [ 2264.553988] Lustre: Skipped 6 previous similar messages [ 2268.348663] LustreError: 138997:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2268.350538] LustreError: 138997:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2268.375900] Lustre: server umount lustre-MDT0000 complete [ 2269.501073] LustreError: 137859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678078 with bad export cookie 7525060209484045996 [ 2269.503667] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2269.506972] LustreError: 137859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2269.645201] Lustre: server umount lustre-MDT0001 complete [ 2281.062906] Lustre: server umount lustre-OST0000 complete [ 2283.857665] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:08:13 (1756678093) [ 2285.956711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2303.455633] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2309.599492] LustreError: 140044:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 2340.036783] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2341.784612] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2343.907282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.050415] LustreError: 140071:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.095802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2345.255166] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2348.121648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2349.585614] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2351.240954] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2352.386763] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2354.526337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2354.554823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2354.608820] Lustre: lustre-OST0001: new disk, initializing [ 2354.610472] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2356.446877] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2356.550568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2356.553491] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2356.599337] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2359.415984] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2360.664779] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2360.730636] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2361.962213] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2362.036137] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2363.872372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2363.877181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.882149] Lustre: Skipped 1 previous similar message [ 2363.884317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2368.748081] LustreError: 143773:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2368.749766] LustreError: 143773:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2368.766612] Lustre: server umount lustre-OST0000 complete [ 2372.064516] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2372.066293] LustreError: 140060:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.066537] LustreError: Skipped 2 previous similar messages [ 2372.070682] LustreError: 140060:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2384.351166] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2384.430902] Lustre: server umount lustre-OST0001 complete [ 2391.826901] Lustre: server umount lustre-MDT0000 complete [ 2392.544344] LustreError: 140974:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.551218] LustreError: 140974:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 2393.055523] LustreError: 142590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678202 with bad export cookie 7525060209484046941 [ 2393.060360] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2393.061753] LustreError: 142590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2393.212594] Lustre: server umount lustre-MDT0001 complete [ 2394.980369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2396.240737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2397.569485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2398.901789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2400.877638] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:10:10 (1756678210) [ 2401.387671] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2401.890371] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2402.395861] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:10:12 (1756678212) [ 2404.463542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2404.548180] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2404.557338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2404.625230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.627199] Lustre: Skipped 6 previous similar messages [ 2405.840554] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2408.590703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2408.624510] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2408.633663] Lustre: 146014:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2409.983555] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2411.721985] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2412.820132] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2414.977446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2415.055601] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2417.191084] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2417.249820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2419.034215] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2420.534456] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2420.621130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2422.029671] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2422.107159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2422.752985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2422.754046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.758039] Lustre: Skipped 5 previous similar messages [ 2428.846828] Lustre: server umount lustre-OST0000 complete [ 2442.821459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2444.725504] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2446.522285] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2447.920205] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2448.035915] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2448.039530] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2453.156795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2453.160380] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2454.124150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 2455.427887] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2455.495767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2456.788348] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f71c9350000.ost_server_uuid 50 [ 2457.325127] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f71c9350000.ost_server_uuid in FULL state after 0 sec [ 2457.380895] LustreError: 146050:0:(ldlm_lockd.c:728:ldlm_handle_ast_error()) ### client (nid 192.168.203.57@tcp) returned error from blocking AST (req@00000000342d0dc7 x1842010147442304 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000008489c9e5/0x686e62641bb92ba2 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.57@tcp remote: 0x9929cebdf46d14ee expref: 5 pid: 146021 timeout: 2557 lvb_type: 0 [ 2457.393785] LustreError: lustre-MDT0000: A client on nid 192.168.203.57@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2457.396669] LustreError: 146010:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.57@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000008489c9e5/0x686e62641bb92ba2 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.57@tcp remote: 0x9929cebdf46d14ee expref: 6 pid: 146021 timeout: 0 lvb_type: 0 [ 2458.596612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2458.598394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2458.598817] LustreError: Skipped 2 previous similar messages [ 2458.600481] Lustre: Skipped 14 previous similar messages [ 2464.638230] Lustre: server umount lustre-OST0000 complete [ 2472.219039] Lustre: server umount lustre-MDT0000 complete [ 2473.496357] LustreError: 147654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678282 with bad export cookie 7525060209484048117 [ 2473.502188] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2479.705061] Lustre: server umount lustre-MDT0001 complete [ 2483.524932] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2484.660211] Key type lgssc unregistered [ 2484.797435] LNet: 151544:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.799700] LNetError: 151544:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2484.807451] LNet: Removed LNI 192.168.203.157@tcp [ 2485.135956] Key type .llcrypt unregistered [ 2485.137041] Key type ._llcrypt unregistered [ 2494.069279] Key type ._llcrypt registered [ 2494.070191] Key type .llcrypt registered [ 2494.108185] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2494.546093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2494.564202] alg: No test for adler32 (adler32-zlib) [ 2495.416081] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2495.497461] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2497.079098] Key type lgssc registered [ 2497.452065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2499.908735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2499.914955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2501.058844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2502.251751] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2505.009548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2505.132566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2506.365614] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2508.023756] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2509.210253] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2511.389058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2511.502190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2513.258049] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2514.943482] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2525.864846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2532.778182] Lustre: Failing over lustre-MDT0000 [ 2532.901887] LustreError: 155181:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2532.930783] Lustre: server umount lustre-MDT0000 complete [ 2535.903868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2535.906605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.910495] LustreError: 152914:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.917827] LustreError: 152914:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2536.419099] LustreError: 152914:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.419350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.423270] LustreError: 152914:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2536.426451] Lustre: Skipped 1 previous similar message [ 2541.537377] LustreError: 152914:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2541.545891] LustreError: 152914:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2543.746219] LustreError: 152896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.57@tcp arrived at 1756678353 with bad export cookie 15635663019022211933 [ 2543.749942] LustreError: 152896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2546.071735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.107377] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2546.193885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.217055] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2547.481386] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2548.202530] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2548.205184] Lustre: lustre-MDT0000: Denying connection for new client af52bb45-8ec2-45e2-8c97-57ce759a9894 (at 192.168.203.57@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2551.269536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2556.386556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.387672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2556.389591] Lustre: Skipped 1 previous similar message [ 2556.400277] Lustre: Skipped 1 previous similar message [ 2561.069378] LustreError: 156109:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2561.071322] LustreError: 156109:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2561.127841] Lustre: server umount lustre-OST0000 complete [ 2562.416972] LustreError: 156314:0:(ldlm_lib.c:2935:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2562.419225] Lustre: 155683:0:(ldlm_lib.c:2338:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2562.428435] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2562.439540] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2562.443439] LustreError: 155683:0:(client.c:1371:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000094baf057 x1842010709329280/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2562.448101] LustreError: 155683:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2562.450773] LustreError: 155683:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2562.453529] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2562.524724] LustreError: 156314:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2562.526416] LustreError: 156314:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2562.552131] Lustre: server umount lustre-MDT0000 complete [ 2563.696181] LustreError: 152895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678373 with bad export cookie 15635663019022212297 [ 2563.700254] LustreError: 152914:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.700632] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2569.795689] LustreError: 156516:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2569.797616] LustreError: 156516:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2569.881700] Lustre: server umount lustre-MDT0001 complete [ 2573.108317] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2574.059825] Key type lgssc unregistered [ 2574.185335] LNet: 157047:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2574.189118] LNetError: 157047:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2574.197340] LNet: Removed LNI 192.168.203.157@tcp [ 2574.482958] Key type .llcrypt unregistered [ 2574.483959] Key type ._llcrypt unregistered [ 2583.146328] Key type ._llcrypt registered [ 2583.147330] Key type .llcrypt registered [ 2583.179398] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2583.452424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2583.459450] alg: No test for adler32 (adler32-zlib) [ 2584.301916] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2584.373152] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2585.943127] Key type lgssc registered [ 2586.264960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2588.625123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2588.631051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.779724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.922764] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2593.593557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2593.715270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2595.004757] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2596.688096] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2597.936572] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2600.076694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2600.182554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2601.888651] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2603.556546] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2615.531533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2620.898366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.899194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2620.899706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2620.901542] Lustre: Skipped 1 previous similar message [ 2622.381805] LustreError: 160683:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2622.440898] Lustre: server umount lustre-OST0000 complete [ 2624.480242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2624.482349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2624.486130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2624.487968] Lustre: Skipped 1 previous similar message [ 2625.723926] LustreError: 160884:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2625.726963] LustreError: 160884:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2625.759326] Lustre: server umount lustre-MDT0000 complete [ 2626.017837] LustreError: 159140:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.018128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.021723] LustreError: 159140:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2626.944259] LustreError: 160584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678436 with bad export cookie 6625856388787317476 [ 2626.949345] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2627.014314] LustreError: 161084:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2627.016112] LustreError: 161084:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2627.099083] Lustre: server umount lustre-MDT0001 complete [ 2630.592994] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2631.660729] Key type lgssc unregistered [ 2631.796520] LNet: 161615:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2631.799898] LNetError: 161615:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2631.806410] LNet: Removed LNI 192.168.203.157@tcp [ 2632.125415] Key type .llcrypt unregistered [ 2632.126466] Key type ._llcrypt unregistered [ 2643.603178] Key type ._llcrypt registered [ 2643.604231] Key type .llcrypt registered [ 2643.646278] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2644.118523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2644.127903] alg: No test for adler32 (adler32-zlib) [ 2644.975096] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2645.067031] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2646.647233] Key type lgssc registered [ 2647.058379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2649.510138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2649.515894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.656103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.836139] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2654.479304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2654.607880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2655.849876] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2657.541603] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2658.688677] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2660.750384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2660.854416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.599962] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2664.283693] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2675.174853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2684.383849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2684.386025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2684.389767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2685.410511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2685.411791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2685.411923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.416954] Lustre: Skipped 1 previous similar message [ 2687.020569] LustreError: 165542:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2687.077541] Lustre: server umount lustre-OST0000 complete [ 2690.528847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.528914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2690.533828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.536522] Lustre: Skipped 1 previous similar message [ 2694.334312] LustreError: 165746:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2694.336795] LustreError: 165746:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2694.362494] Lustre: server umount lustre-MDT0000 complete [ 2695.531968] LustreError: 165747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678504 with bad export cookie 16135116869261782579 [ 2695.538103] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2695.598449] LustreError: 165949:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2695.600135] LustreError: 165949:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2695.678306] Lustre: server umount lustre-MDT0001 complete [ 2698.951098] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2699.992434] Key type lgssc unregistered [ 2700.126545] LNet: 166480:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2700.128573] LNetError: 166480:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2700.136455] LNet: Removed LNI 192.168.203.157@tcp [ 2700.433914] Key type .llcrypt unregistered [ 2700.435028] Key type ._llcrypt unregistered [ 2708.859581] Key type ._llcrypt registered [ 2708.860599] Key type .llcrypt registered [ 2708.897912] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2709.346150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2709.356445] alg: No test for adler32 (adler32-zlib) [ 2710.209341] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2710.296418] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2711.879220] Key type lgssc registered [ 2712.244962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2715.725826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2715.733705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.853516] Lustre: *** cfs_fail_loc=135, val=0*** [ 2716.890859] LustreError: 167961:0:(obd_config.c:782:class_setup()) setup lustre-MDT0000 failed (-2) [ 2716.892969] LustreError: 167961:0:(obd_config.c:1996:class_config_llog_handler()) MGC192.168.203.157@tcp: cfg command failed: rc = -2 [ 2716.895498] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2716.895498] [ 2716.898259] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2716.903771] LustreError: 167921:0:(tgt_mount.c:1710:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2716.907097] LustreError: 167921:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -2 [ 2716.910288] LustreError: 167921:0:(obd_config.c:835:class_cleanup()) Device 5 not setup [ 2716.913449] LustreError: 167921:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2716.969816] Lustre: server umount lustre-MDT0000 complete [ 2716.971324] LustreError: 167921:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2719.235042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2719.413481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2720.716868] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2723.405590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2723.524528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2724.740654] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2726.459733] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2727.573506] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2728.195241] LustreError: 168362:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2728.196904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.197820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2728.199594] LustreError: 168362:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2728.202707] Lustre: Skipped 1 previous similar message [ 2728.208131] LustreError: 168362:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2733.344262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2733.345965] Lustre: Skipped 2 previous similar messages [ 2734.329652] LustreError: 169674:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2734.333151] LustreError: 169674:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2734.357883] Lustre: server umount lustre-MDT0000 complete [ 2735.529105] LustreError: 169677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678544 with bad export cookie 9784381022653328269 [ 2735.532139] LustreError: 169677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2735.532647] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2735.586646] LustreError: 169879:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2735.588478] LustreError: 169879:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2735.657607] Lustre: server umount lustre-MDT0001 complete [ 2739.234436] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2740.218334] Key type lgssc unregistered [ 2740.345318] LNet: 170410:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2740.347605] LNetError: 170410:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2740.353298] LNet: Removed LNI 192.168.203.157@tcp [ 2740.639937] Key type .llcrypt unregistered [ 2740.640914] Key type ._llcrypt unregistered [ 2747.377768] Key type ._llcrypt registered [ 2747.378819] Key type .llcrypt registered [ 2747.412925] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2747.502647] Key type .llcrypt unregistered [ 2747.503672] Key type ._llcrypt unregistered [ 2754.613504] Key type ._llcrypt registered [ 2754.614699] Key type .llcrypt registered [ 2754.666183] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2755.150370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2755.157802] alg: No test for adler32 (adler32-zlib) [ 2756.011511] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2756.103145] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2757.695154] Key type lgssc registered [ 2758.000432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2760.378840] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2760.384672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2777.887217] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2783.007201] LustreError: 172476:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 2813.727124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2815.378568] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2817.040543] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2819.197149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.363130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.387308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2820.579472] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2823.249571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2823.384958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2824.734785] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2826.445687] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2827.545271] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2828.456465] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2829.792092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2829.794184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.797647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2833.888715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2833.891600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2833.891629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2835.438131] LustreError: 174962:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2835.508259] Lustre: server umount lustre-OST0000 complete [ 2839.009139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2839.009327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.009780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2839.009784] Lustre: Skipped 1 previous similar message [ 2839.017574] Lustre: Skipped 1 previous similar message [ 2844.128187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2844.129911] Lustre: Skipped 2 previous similar messages [ 2849.248152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2849.250423] Lustre: Skipped 1 previous similar message [ 2851.295187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2851.333656] LustreError: 175164:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2851.335379] LustreError: 175164:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2851.360730] Lustre: server umount lustre-MDT0000 complete [ 2852.563294] LustreError: 175165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678661 with bad export cookie 17808246592292148766 [ 2852.566543] LustreError: 175165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2852.567177] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2852.628962] LustreError: 175367:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2852.630857] LustreError: 175367:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2852.716633] Lustre: server umount lustre-MDT0001 complete [ 2856.322742] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2857.378289] Key type lgssc unregistered [ 2857.515409] LNet: 175898:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2857.519168] LNetError: 175898:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2857.529337] LNet: Removed LNI 192.168.203.157@tcp [ 2857.829676] Key type .llcrypt unregistered [ 2857.831037] Key type ._llcrypt unregistered [ 2866.135278] Key type ._llcrypt registered [ 2866.136206] Key type .llcrypt registered [ 2866.167454] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2866.454701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2866.466295] alg: No test for adler32 (adler32-zlib) [ 2867.315955] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2867.407699] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2868.991131] Key type lgssc registered [ 2869.335726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2871.849922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2871.855816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2873.014700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2874.266408] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2877.018795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2877.131850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2878.363479] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2880.036370] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2881.258032] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2883.454526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2883.563841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.374535] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2887.093214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 2887.154261] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2889.193974] Lustre: Failing over lustre-MDT0000 [ 2889.265228] LustreError: 179566:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2889.294940] Lustre: server umount lustre-MDT0000 complete [ 2892.767981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2892.771872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.778607] LustreError: 179431:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2892.782974] LustreError: 179431:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2893.284311] LustreError: 177270:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.284870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.292062] LustreError: 177270:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2893.295769] Lustre: Skipped 1 previous similar message [ 2898.400437] LustreError: 177270:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.405024] LustreError: 177270:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 2902.001685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.040573] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2902.125310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2902.143036] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2903.314247] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2903.933154] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2904.812798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 2907.627895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2907.633503] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2907.647740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 2909.280406] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 2914.785414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2914.788257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2914.793951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2917.857898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2917.861228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.557126] LustreError: 180843:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2920.559484] LustreError: 180843:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2920.627492] Lustre: server umount lustre-OST0000 complete [ 2922.976909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2922.977080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2922.982940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2922.985073] Lustre: Skipped 2 previous similar messages [ 2928.096036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2928.098851] Lustre: Skipped 1 previous similar message [ 2933.216097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2933.217679] Lustre: Skipped 1 previous similar message [ 2936.287169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2936.326239] LustreError: 181048:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2936.332949] LustreError: 181048:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2936.368725] Lustre: server umount lustre-MDT0000 complete [ 2937.613694] LustreError: 177251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678746 with bad export cookie 14567698096145385114 [ 2937.617383] LustreError: 177251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2937.618388] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2937.776449] Lustre: server umount lustre-MDT0001 complete [ 2941.062501] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2942.027518] Key type lgssc unregistered [ 2942.157337] LNet: 181781:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2942.159423] LNetError: 181781:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2942.168271] LNet: Removed LNI 192.168.203.157@tcp [ 2942.452758] Key type .llcrypt unregistered [ 2942.453802] Key type ._llcrypt unregistered [ 2951.183343] Key type ._llcrypt registered [ 2951.184519] Key type .llcrypt registered [ 2951.220708] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2951.616182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2951.625132] alg: No test for adler32 (adler32-zlib) [ 2952.480941] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2952.561857] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2954.143126] Key type lgssc registered [ 2954.528106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2957.088629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2957.095210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2958.245526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.409674] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2961.986509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2962.104458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2963.300548] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2964.992612] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2966.243328] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2968.332288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2968.445425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2969.511708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 2970.162556] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2971.769861] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2973.277160] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 2979.768586] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 2991.037331] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3000.288224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3000.290645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.294691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3003.360455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3003.362521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3003.364240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.128989] LustreError: 185584:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3006.196529] Lustre: server umount lustre-OST0000 complete [ 3008.482565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3008.482806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3008.491199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3008.492979] Lustre: Skipped 2 previous similar messages [ 3013.501624] LustreError: 185789:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3013.503394] LustreError: 185789:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3013.532778] Lustre: server umount lustre-MDT0000 complete [ 3013.601587] LustreError: 183153:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.605695] LustreError: 183153:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3014.689111] LustreError: 185350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678823 with bad export cookie 5277078024021564906 [ 3014.692709] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3014.694334] LustreError: 185350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3014.753447] LustreError: 185992:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3014.755242] LustreError: 185992:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3014.828872] Lustre: server umount lustre-MDT0001 complete [ 3018.218632] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3019.313263] Key type lgssc unregistered [ 3019.457516] LNet: 186523:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3019.459619] LNetError: 186523:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3019.467332] LNet: Removed LNI 192.168.203.157@tcp [ 3019.773744] Key type .llcrypt unregistered [ 3019.774874] Key type ._llcrypt unregistered [ 3028.738393] Key type ._llcrypt registered [ 3028.739523] Key type .llcrypt registered [ 3028.773067] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3029.124674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3029.152157] alg: No test for adler32 (adler32-zlib) [ 3030.000183] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3030.081735] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3031.655127] Key type lgssc registered [ 3031.979542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3034.322341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3034.328284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3035.472351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3036.687769] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3039.846359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3039.965093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3041.132124] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3042.812534] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3043.949651] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3046.029685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3046.147591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.981049] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3049.631394] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3060.518599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3066.567278] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3076.745824] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3086.758894] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3095.112615] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3106.785975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3106.788210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3106.791438] Lustre: Skipped 1 previous similar message [ 3106.791868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3106.794334] Lustre: Skipped 1 previous similar message [ 3111.726150] LustreError: 192276:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3111.786586] Lustre: server umount lustre-OST0000 complete [ 3117.024789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.024894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3117.025483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3117.028683] Lustre: Skipped 1 previous similar message [ 3117.033996] LustreError: Skipped 1 previous similar message [ 3119.225984] LustreError: 192477:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3119.228155] LustreError: 192477:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3119.254392] Lustre: server umount lustre-MDT0000 complete [ 3120.482482] LustreError: 190804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678929 with bad export cookie 14077088326647241109 [ 3120.485324] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3120.486744] LustreError: 190804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3120.549179] LustreError: 192678:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3120.551109] LustreError: 192678:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3120.627051] Lustre: server umount lustre-MDT0001 complete [ 3124.375688] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3125.332077] Key type lgssc unregistered [ 3125.460293] LNet: 193210:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3125.462602] LNetError: 193210:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3125.468296] LNet: Removed LNI 192.168.203.157@tcp [ 3125.748343] Key type .llcrypt unregistered [ 3125.749597] Key type ._llcrypt unregistered [ 3135.468579] Key type ._llcrypt registered [ 3135.470183] Key type .llcrypt registered [ 3135.519671] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3135.927858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3135.969760] alg: No test for adler32 (adler32-zlib) [ 3136.862291] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3136.962476] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3138.567169] Key type lgssc registered [ 3138.990799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3141.832855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3141.842256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3143.008804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3144.441500] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3147.751544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3147.892088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3149.284072] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3151.482398] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3152.631477] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3155.135495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3155.320813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3157.631944] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3159.734750] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3168.617310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3178.976901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3178.978078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3178.979468] LustreError: Skipped 1 previous similar message [ 3178.983320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3178.985902] Lustre: Skipped 1 previous similar message [ 3180.076763] LustreError: 196996:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3180.133550] Lustre: server umount lustre-OST0000 complete [ 3183.583815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3183.586305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3183.590079] Lustre: Skipped 1 previous similar message [ 3183.591768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3187.514970] LustreError: 197200:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3187.517050] LustreError: 197200:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3187.543057] Lustre: server umount lustre-MDT0000 complete [ 3188.847295] LustreError: 194560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678998 with bad export cookie 11963208097034438391 [ 3188.853668] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3188.857169] LustreError: 194560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3188.956615] LustreError: 197401:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3188.959659] LustreError: 197401:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3189.058181] Lustre: server umount lustre-MDT0001 complete [ 3192.992149] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3194.150758] Key type lgssc unregistered [ 3194.299413] LNet: 197932:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3194.301732] LNetError: 197932:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3194.310408] LNet: Removed LNI 192.168.203.157@tcp [ 3194.651802] Key type .llcrypt unregistered [ 3194.652978] Key type ._llcrypt unregistered [ 3202.685946] Key type ._llcrypt registered [ 3202.687071] Key type .llcrypt registered [ 3202.731562] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3203.246429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3203.252505] alg: No test for adler32 (adler32-zlib) [ 3204.148130] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3204.239926] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3205.823218] Key type lgssc registered [ 3206.351310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3209.284190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3209.294190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.478368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3211.780515] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3214.664380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3214.795673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3216.167713] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3218.203779] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3219.401980] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3221.750891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3221.868249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3223.583012] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3225.490216] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3235.174297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3240.422515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3240.422703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.425068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3240.429238] Lustre: Skipped 1 previous similar message [ 3241.647152] LustreError: 201581:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3241.719079] Lustre: server umount lustre-OST0000 complete [ 3245.119857] LustreError: 201782:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3245.121581] LustreError: 201782:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3245.148446] Lustre: server umount lustre-MDT0000 complete [ 3245.536144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3245.538238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.539629] LustreError: Skipped 1 previous similar message [ 3245.546052] LustreError: 199152:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3245.551390] LustreError: 199152:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3246.488079] LustreError: 199139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679055 with bad export cookie 5465700543692081747 [ 3246.492253] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3246.492835] LustreError: 199139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3246.559845] LustreError: 201985:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3246.561654] LustreError: 201985:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3246.637828] Lustre: server umount lustre-MDT0001 complete [ 3250.400709] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3251.570269] Key type lgssc unregistered [ 3251.716497] LNet: 202516:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3251.720909] LNetError: 202516:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3251.733454] LNet: Removed LNI 192.168.203.157@tcp [ 3252.033875] Key type .llcrypt unregistered [ 3252.034794] Key type ._llcrypt unregistered [ 3259.235552] Key type ._llcrypt registered [ 3259.236949] Key type .llcrypt registered [ 3259.277479] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3259.589904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3259.623397] alg: No test for adler32 (adler32-zlib) [ 3260.465793] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3260.541374] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3262.119235] Key type lgssc registered [ 3262.671149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3265.349141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3265.356512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.502968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3267.687185] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3270.334436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3270.461996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3271.685444] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3273.806247] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3275.375921] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3278.087955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3278.252027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.238213] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3282.089088] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3292.582904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3300.097338] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3300.121435] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3300.210954] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3302.093584] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3303.830526] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3309.459484] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 3320.169241] Lustre: Permanently deactivating lustre-OST0001 [ 3320.182740] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3329.527230] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3333.470058] Lustre: Permanently reactivating lustre-OST0001 [ 3333.475528] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3333.477386] Lustre: Skipped 2 previous similar messages [ 3336.001448] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3336.006996] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3336.009529] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3336.013864] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3336.029068] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3343.846994] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.849970] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3343.853376] Lustre: Skipped 2 previous similar messages [ 3343.859543] Lustre: Skipped 1 previous similar message [ 3348.082116] LustreError: 207253:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3348.106342] Lustre: server umount lustre-OST0001 complete [ 3348.966512] LustreError: 205699:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3348.977679] LustreError: 205699:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3352.032688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3352.037088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.045667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3354.082257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3354.083677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.087201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3354.087296] LustreError: 205699:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.202603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3359.206962] Lustre: Skipped 2 previous similar messages [ 3359.207041] LustreError: 205699:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.218715] LustreError: 205699:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3364.319109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3364.326489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3364.326975] LustreError: 205336:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3364.328799] Lustre: Skipped 1 previous similar message [ 3364.333149] LustreError: 205336:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3364.353200] LustreError: 207455:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3364.354986] LustreError: 207455:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3364.443719] Lustre: server umount lustre-OST0000 complete [ 3367.904405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3367.908660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3374.561103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3374.565695] Lustre: Skipped 4 previous similar messages [ 3380.191132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3380.245904] LustreError: 207658:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3380.249768] LustreError: 207658:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3380.288922] Lustre: server umount lustre-MDT0000 complete [ 3381.781540] LustreError: 203719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679191 with bad export cookie 10755558162801587662 [ 3381.787166] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3381.789800] LustreError: 203719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3381.984255] Lustre: server umount lustre-MDT0001 complete [ 3386.266214] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3387.638511] Key type lgssc unregistered [ 3387.797600] LNet: 208392:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3387.802306] LNetError: 208392:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3387.810471] LNet: Removed LNI 192.168.203.157@tcp [ 3388.160651] Key type .llcrypt unregistered [ 3388.161791] Key type ._llcrypt unregistered [ 3400.054091] Key type ._llcrypt registered [ 3400.055205] Key type .llcrypt registered [ 3400.096948] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 3406.162555] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3406.674708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3406.693425] alg: No test for adler32 (adler32-zlib) [ 3407.550879] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3407.645879] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3409.231177] Key type lgssc registered [ 3409.644496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3412.470678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3414.429324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3416.264301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3418.192152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3420.346805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3420.372214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3420.379406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.469944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3421.480291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3421.518436] Lustre: lustre-MDT0000: new disk, initializing [ 3421.544195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.550697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3422.823675] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3426.780456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3426.808547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3426.839543] Lustre: 212310:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3426.864910] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3426.866528] Lustre: Skipped 1 previous similar message [ 3426.904471] Lustre: lustre-MDT0001: new disk, initializing [ 3426.937481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3426.948154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3426.951588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3428.213627] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3431.107725] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3432.180202] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3434.259863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3434.281811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3434.367805] Lustre: lustre-OST0000: new disk, initializing [ 3434.369490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3434.392757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3434.860125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3434.863957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3434.896021] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3436.138771] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3439.181186] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3440.460987] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3440.529296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3441.755147] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3441.828673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3445.219417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3445.221494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3445.221500] LustreError: Skipped 1 previous similar message [ 3445.228376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3445.229990] Lustre: Skipped 1 previous similar message [ 3448.557117] LustreError: 215249:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3448.624746] Lustre: server umount lustre-OST0000 complete [ 3450.337991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3450.339101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3450.341022] Lustre: Skipped 2 previous similar messages [ 3455.456952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3455.461103] Lustre: Skipped 2 previous similar messages [ 3460.577867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3460.582289] Lustre: Skipped 1 previous similar message [ 3464.159133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3464.197451] LustreError: 215450:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3464.200661] LustreError: 215450:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3464.232427] Lustre: server umount lustre-MDT0000 complete [ 3465.749469] LustreError: 212302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679275 with bad export cookie 6105819408455931739 [ 3465.754072] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3465.823836] LustreError: 215653:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3465.825797] LustreError: 215653:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3465.925232] Lustre: server umount lustre-MDT0001 complete [ 3469.000287] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 18:27:58 (1756679278) [ 3471.206643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3471.349785] LustreError: 216323:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3471.354856] LustreError: 216323:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3471.370118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.596410] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3475.584341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3475.691868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3476.902800] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3478.467527] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3479.585981] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3481.640601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3483.490465] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3485.100852] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3501.499571] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3511.086200] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3517.584185] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3527.153611] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3535.768141] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3556.137718] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3556.142131] Lustre: Skipped 1 previous similar message [ 3588.699207] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3588.701563] Lustre: Skipped 4 previous similar messages [ 3661.840072] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 3661.842776] Lustre: Skipped 9 previous similar messages [ 3664.353232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3664.358091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3664.364285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3665.383960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3669.547947] LustreError: 219830:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3669.549953] LustreError: 219830:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3669.606405] Lustre: server umount lustre-OST0000 complete [ 3675.616582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3675.616676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3675.617385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3675.617389] Lustre: Skipped 2 previous similar messages [ 3675.624995] Lustre: Skipped 2 previous similar messages [ 3685.343086] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3685.371771] LustreError: 220035:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3685.373537] LustreError: 220035:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3685.399509] Lustre: server umount lustre-MDT0000 complete [ 3685.856766] LustreError: 216322:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3685.866131] LustreError: 216322:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3686.941513] LustreError: 219619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679496 with bad export cookie 6105819408455932607 [ 3686.946482] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3686.952504] LustreError: 219619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3687.141302] Lustre: server umount lustre-MDT0001 complete [ 3691.395167] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3692.835930] Key type lgssc unregistered [ 3692.976653] LNet: 220768:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3692.983389] LNetError: 220768:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3692.994569] LNet: Removed LNI 192.168.203.157@tcp [ 3693.406517] Key type .llcrypt unregistered [ 3693.408133] Key type ._llcrypt unregistered [ 3703.520771] Key type ._llcrypt registered [ 3703.522237] Key type .llcrypt registered [ 3703.563657] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3703.873498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3703.911648] alg: No test for adler32 (adler32-zlib) [ 3704.780146] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3704.859099] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3706.439188] Key type lgssc registered [ 3707.032434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3710.368075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3710.375509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.555478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3713.042814] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3716.586543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3716.770423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3718.266057] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3720.484997] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3722.123345] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3725.095400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3725.227534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3727.640873] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3729.942338] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3747.809447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3747.817803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3747.830696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3751.908773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3751.916535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3751.924666] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3753.648608] LustreError: 224572:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3753.742293] Lustre: server umount lustre-OST0000 complete [ 3757.026226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3757.027849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3757.034735] Lustre: Skipped 1 previous similar message [ 3757.038236] Lustre: Skipped 1 previous similar message [ 3761.739072] LustreError: 224777:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3761.743566] LustreError: 224777:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3761.779343] Lustre: server umount lustre-MDT0000 complete [ 3762.145354] LustreError: 222153:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3762.155779] LustreError: 222153:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3763.483218] LustreError: 224474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679572 with bad export cookie 8334774950571999975 [ 3763.489164] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3763.490117] LustreError: 224474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3763.591452] LustreError: 224978:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3763.595170] LustreError: 224978:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3763.684409] Lustre: server umount lustre-MDT0001 complete [ 3768.219496] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3769.440711] Key type lgssc unregistered [ 3769.562318] LNet: 225510:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3769.565122] LNetError: 225510:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3769.574294] LNet: Removed LNI 192.168.203.157@tcp [ 3769.854608] Key type .llcrypt unregistered [ 3769.856058] Key type ._llcrypt unregistered [ 3778.227120] Key type ._llcrypt registered [ 3778.228293] Key type .llcrypt registered [ 3778.267658] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3778.404226] Key type .llcrypt unregistered [ 3778.405903] Key type ._llcrypt unregistered [ 3785.706983] Key type ._llcrypt registered [ 3785.708162] Key type .llcrypt registered [ 3785.739085] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 3793.174458] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3793.629520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3793.642328] alg: No test for adler32 (adler32-zlib) [ 3794.532192] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3794.631974] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3796.215179] Key type lgssc registered [ 3796.557534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3800.774291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3803.954582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3806.950835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3810.012783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3810.921634] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3668 sec ========== 18:33:40 (1756679620) [ 3811.677598] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:33:41 (1756679621) === [ 3813.042406] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:33:42 (1756679622) === [ 3830.312395] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3831.595665] Key type lgssc unregistered [ 3831.768674] LNet: 231591:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3831.773221] LNetError: 231591:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3831.783731] LNet: Removed LNI 192.168.203.157@tcp [ 3832.284446] Key type .llcrypt unregistered [ 3832.285806] Key type ._llcrypt unregistered