[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 465780912 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.003107] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.013019] SELinux: Initializing. [ 0.014084] *** VALIDATE selinux *** [ 0.022112] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028303] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031111] *** VALIDATE tmpfs *** [ 0.033443] *** VALIDATE proc *** [ 0.035190] *** VALIDATE cgroup *** [ 0.035936] *** VALIDATE cgroup2 *** [ 0.036189] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037131] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039024] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.042630] debug: unmapping init [mem 0xffffffffb0c59000-0xffffffffb0c60fff] [ 0.044705] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045471] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046022] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049009] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053305] rcu: Hierarchical SRCU implementation. [ 0.055376] smp: Bringing up secondary CPUs ... [ 0.056595] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.060251] smp: Brought up 1 node, 4 CPUs [ 0.062010] smpboot: Max logical packages: 1 [ 0.063018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148018] node 0 deferred pages initialised in 82ms [ 0.151214] devtmpfs: initialized [ 0.152522] x86/mm: Memory block size: 128MB [ 0.154000] gcov: version magic: 0x41383552 [ 0.158157] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162252] pinctrl core: initialized pinctrl subsystem [ 0.164187] [ 0.164801] ************************************************************* [ 0.167020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170013] ** ** [ 0.172010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175015] ** ** [ 0.179015] ** This means that this kernel is built to expose internal ** [ 0.181011] ** IOMMU data structures, which may compromise security on ** [ 0.183010] ** your system. ** [ 0.185011] ** ** [ 0.187011] ** If you see this message and you are not debugging the ** [ 0.189020] ** kernel, report this immediately to your vendor! ** [ 0.191010] ** ** [ 0.193011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195013] ************************************************************* [ 0.198607] NET: Registered protocol family 16 [ 0.200545] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.203068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211039] cpuidle: using governor menu [ 0.213533] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.217262] PCI: Using configuration type 1 for base access [ 0.219126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.230322] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.233038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.237332] cryptd: max_cpu_qlen set to 1000 [ 0.241250] ACPI: Added _OSI(Module Device) [ 0.242021] ACPI: Added _OSI(Processor Device) [ 0.244015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.246015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.249803] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.256636] ACPI: Interpreter enabled [ 0.257136] ACPI: PM: (supports S0 S3 S4 S5) [ 0.259012] ACPI: Using IOAPIC for interrupt routing [ 0.261123] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.264476] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.273878] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.278054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.280019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.284098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.290322] acpiphp: Slot [2] registered [ 0.292135] acpiphp: Slot [5] registered [ 0.294114] acpiphp: Slot [6] registered [ 0.295125] acpiphp: Slot [7] registered [ 0.297151] acpiphp: Slot [8] registered [ 0.300146] acpiphp: Slot [9] registered [ 0.301104] acpiphp: Slot [10] registered [ 0.303133] acpiphp: Slot [3] registered [ 0.304114] acpiphp: Slot [4] registered [ 0.306115] acpiphp: Slot [11] registered [ 0.308088] acpiphp: Slot [12] registered [ 0.309125] acpiphp: Slot [13] registered [ 0.311106] acpiphp: Slot [14] registered [ 0.313106] acpiphp: Slot [15] registered [ 0.314098] acpiphp: Slot [16] registered [ 0.316087] acpiphp: Slot [17] registered [ 0.318141] acpiphp: Slot [18] registered [ 0.320106] acpiphp: Slot [19] registered [ 0.322112] acpiphp: Slot [20] registered [ 0.323111] acpiphp: Slot [21] registered [ 0.325119] acpiphp: Slot [22] registered [ 0.327115] acpiphp: Slot [23] registered [ 0.329201] acpiphp: Slot [24] registered [ 0.331111] acpiphp: Slot [25] registered [ 0.332099] acpiphp: Slot [26] registered [ 0.334112] acpiphp: Slot [27] registered [ 0.336120] acpiphp: Slot [28] registered [ 0.337111] acpiphp: Slot [29] registered [ 0.339133] acpiphp: Slot [30] registered [ 0.341102] acpiphp: Slot [31] registered [ 0.343069] PCI host bridge to bus 0000:00 [ 0.344019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.347032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.350034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.353041] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.356024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.359026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.361175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.364011] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.368299] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.378014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.385045] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.388020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.391020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.393017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.397382] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.400811] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.404047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.407883] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.413016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.423718] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.428015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.435000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.448015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.456016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.475012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.492439] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.501018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.507017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.527019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.538268] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.545016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.551015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.574019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.585719] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.597016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.603012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.620014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.629808] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.636014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.643016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.666015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.677402] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.681985] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.688015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.711023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.722680] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.725463] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.729386] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.731410] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.734255] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.739118] iommu: Default domain type: Passthrough [ 0.741636] SCSI subsystem initialized [ 0.743202] ACPI: bus type USB registered [ 0.745153] usbcore: registered new interface driver usbfs [ 0.748148] usbcore: registered new interface driver hub [ 0.750108] usbcore: registered new device driver usb [ 0.751276] pps_core: LinuxPPS API ver. 1 registered [ 0.753017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.757087] PTP clock support registered [ 0.760063] EDAC MC: Ver: 3.0.0 [ 0.762115] PCI: Using ACPI for IRQ routing [ 0.765413] NetLabel: Initializing [ 0.766013] NetLabel: domain hash size = 128 [ 0.768014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.770080] NetLabel: unlabeled traffic allowed by default [ 0.772164] vgaarb: loaded [ 0.775354] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.776016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.782706] clocksource: Switched to clocksource kvm-clock [ 0.896584] VFS: Disk quotas dquot_6.6.0 [ 0.898865] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.902516] *** VALIDATE ramfs *** [ 0.904124] *** VALIDATE hugetlbfs *** [ 0.906206] pnp: PnP ACPI init [ 0.909067] pnp: PnP ACPI: found 6 devices [ 0.936696] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.940136] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.943046] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.945643] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.948102] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.951020] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.954426] NET: Registered protocol family 2 [ 0.957633] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.963496] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.967508] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.973693] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.977774] TCP: Hash tables configured (established 65536 bind 65536) [ 0.980314] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.982762] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.985983] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.989311] NET: Registered protocol family 1 [ 0.993378] RPC: Registered named UNIX socket transport module. [ 0.996323] RPC: Registered udp transport module. [ 0.998519] RPC: Registered tcp transport module. [ 1.000495] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.003050] NET: Registered protocol family 44 [ 1.005035] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.006879] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.008673] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.010651] PCI: CLS 0 bytes, default 64 [ 1.012154] Unpacking initramfs... [ 2.479090] debug: unmapping init [mem 0xffff8e22fcc54000-0xffff8e22fffbffff] [ 2.485965] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.488498] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.493180] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.992427] Initialise system trusted keyrings [ 2.994437] Key type blacklist registered [ 2.996836] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.007932] zbud: loaded [ 3.011492] *** VALIDATE nfs *** [ 3.013058] *** VALIDATE nfs4 *** [ 3.014944] pstore: using deflate compression [ 3.019425] Platform Keyring initialized [ 3.121050] NET: Registered protocol family 38 [ 3.123085] Key type asymmetric registered [ 3.124569] Asymmetric key parser 'x509' registered [ 3.126360] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.130183] io scheduler mq-deadline registered [ 3.132364] io scheduler kyber registered [ 3.134471] io scheduler bfq registered [ 3.136872] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.141195] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.146117] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.151306] ACPI: Power Button [PWRF] [ 3.156827] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.164669] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.186455] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.196803] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.214533] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.242827] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.271158] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.276472] Non-volatile memory driver v1.3 [ 3.278532] Linux agpgart interface v0.103 [ 3.308582] virtio_blk virtio1: [vda] 146936 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.311782] vda: detected capacity change from 0 to 75231232 [ 3.327968] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.332333] vdb: detected capacity change from 0 to 1073741824 [ 3.353606] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.356545] vdc: detected capacity change from 0 to 2621440000 [ 3.369815] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.373337] vdd: detected capacity change from 0 to 2621440000 [ 3.386733] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.391978] vde: detected capacity change from 0 to 4294967296 [ 3.408467] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.412018] vdf: detected capacity change from 0 to 4294967296 [ 3.423789] libphy: Fixed MDIO Bus: probed [ 3.429508] usbcore: registered new interface driver usbserial_generic [ 3.432988] usbserial: USB Serial support registered for generic [ 3.436043] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.441168] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.443449] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.446525] mousedev: PS/2 mouse device common for all mice [ 3.450255] rtc_cmos 00:05: RTC can wake from S4 [ 3.453980] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.460145] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.464815] rtc_cmos 00:05: registered as rtc0 [ 3.466150] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.466875] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.474229] intel_pstate: CPU model not supported [ 3.477716] hid: raw HID events driver (C) Jiri Kosina [ 3.480140] usbcore: registered new interface driver usbhid [ 3.482340] usbhid: USB HID core driver [ 3.484094] drop_monitor: Initializing network drop monitor service [ 3.486722] Initializing XFRM netlink socket [ 3.488959] NET: Registered protocol family 10 [ 3.492144] Segment Routing with IPv6 [ 3.493310] NET: Registered protocol family 17 [ 3.494957] mpls_gso: MPLS GSO support [ 3.502547] RAS: Correctable Errors collector initialized. [ 3.505074] AVX version of gcm_enc/dec engaged. [ 3.506593] AES CTR mode by8 optimization enabled [ 3.598230] sched_clock: Marking stable (3598204763, 0)->(4514290108, -916085345) [ 3.601215] registered taskstats version 1 [ 3.602984] Loading compiled-in X.509 certificates [ 3.604698] zswap: loaded using pool lzo/zbud [ 3.630632] Key type big_key registered [ 3.644101] Key type encrypted registered [ 3.646492] ima: No TPM chip found, activating TPM-bypass! [ 3.649563] ima: Allocated hash algorithm: sha1 [ 3.651474] ima: No architecture policies found [ 3.653500] evm: Initialising EVM extended attributes: [ 3.655762] evm: security.selinux [ 3.657329] evm: security.ima [ 3.658660] evm: security.capability [ 3.660305] evm: HMAC attrs: 0x1 [ 3.663084] rtc_cmos 00:05: setting system clock to 2026-08-27 14:54:40 UTC (1787842480) [ 3.670845] debug: unmapping init [mem 0xffffffffb1c03000-0xffffffffb1dfffff] [ 3.674589] debug: unmapping init [mem 0xffffffffb0982000-0xffffffffb0c58fff] [ 3.689070] Write protecting the kernel read-only data: 28672k [ 3.693257] debug: unmapping init [mem 0xffffffffaf003000-0xffffffffaf1fffff] [ 3.696568] debug: unmapping init [mem 0xffffffffaf914000-0xffffffffaf9fffff] [ 3.738864] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.750477] systemd[1]: Detected virtualization kvm. [ 3.752692] systemd[1]: Detected architecture x86-64. [ 3.754826] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.783969] systemd[1]: No hostname configured. [ 3.786858] systemd[1]: Set hostname to . [ 3.789243] random: systemd: uninitialized urandom read (16 bytes read) [ 3.791802] systemd[1]: Initializing machine ID from random generator. [ 3.844687] random: ln: uninitialized urandom read (6 bytes read) [ 3.946250] random: systemd: uninitialized urandom read (16 bytes read) [ 3.949300] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.953419] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.957555] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.586531] device-mapper: uevent: version 1.0.3 [ 4.589108] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.256865] virtio_net virtio0 ens2: renamed from eth0 [ 5.265491] random: fast init done [ 5.323839] scsi host0: ata_piix [ 5.344212] scsi host1: ata_piix [ 5.345991] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.348945] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.012733] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.167136] random: crng init done [ 10.168590] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.588216] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.815784] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.061655] SELinux: Disabled at runtime. [ 12.122195] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.132397] systemd[1]: Detected virtualization kvm. [ 12.134596] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.632320] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.635851] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.642452] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.647255] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.653172] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.662076] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.669910] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.800726] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.199714] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.577249] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.665603] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.773241] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.785235] EDAC sbridge: Ver: 1.1.2 [ 15.261811] Key type dns_resolver registered [ 15.593836] NFS: Registering the id_resolver key type [ 15.596402] Key type id_resolver registered [ 15.598049] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg352-server login: [ 54.651317] libcfs: loading out-of-tree module taints kernel. [ 54.695288] Key type ._llcrypt registered [ 54.698151] Key type .llcrypt registered [ 54.814174] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 75.444548] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 75.886922] hrtimer: interrupt took 6942881 ns [ 77.777099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 77.792614] alg: No test for adler32 (adler32-zlib) [ 79.302801] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 80.758931] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 82.553233] Key type lgssc registered [ 84.990921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.004972] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.951280] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 155.970917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 155.987716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.259651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 157.299443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 157.444634] Lustre: lustre-MDT0000: new disk, initializing [ 157.534269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.574296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.857532] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.535852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.671717] Lustre: 6520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.717766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.723202] Lustre: Skipped 1 previous similar message [ 177.824422] Lustre: lustre-MDT0001: new disk, initializing [ 177.891665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.929871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 177.945567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 182.693768] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.965182] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 198.209244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 198.353991] Lustre: lustre-OST0000: new disk, initializing [ 198.356412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 198.359691] Lustre: 8460:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 198.435741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.779488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 203.800163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 203.873430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.963153] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.747625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.919227] Lustre: lustre-OST0001: new disk, initializing [ 220.925186] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 220.932725] Lustre: 9535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.033675] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.605768] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.349907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 227.365829] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 227.470727] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 238.329372] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 245.015266] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 251.395402] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing check_logdir /tmp/testlogs/ [ 256.468917] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing yml_node [ 261.278405] Lustre: DEBUG MARKER: Client: 2.17.57.92 [ 263.918435] Lustre: DEBUG MARKER: MDS: 2.17.57.92 [ 266.331759] Lustre: DEBUG MARKER: OSS: 2.17.57.92 [ 268.020668] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 10:59:02 EDT 2026 [ 282.456762] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 283.712906] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 298.979197] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.982941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 298.988394] Lustre: Skipped 3 previous similar messages [ 299.005403] Lustre: Skipped 3 previous similar messages [ 304.097434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.110030] Lustre: Skipped 3 previous similar messages [ 305.065493] Lustre: server umount lustre-MDT0000 complete [ 308.291759] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842785 with bad export cookie 11492017222315871046 [ 308.295896] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 308.305247] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 309.221321] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.237302] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 309.237641] Lustre: Skipped 1 previous similar message [ 309.253930] Lustre: Skipped 1 previous similar message [ 313.254407] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.263648] Lustre: Skipped 1 previous similar message [ 314.788318] Lustre: server umount lustre-MDT0001 complete [ 318.686049] Lustre: server umount lustre-OST0000 complete [ 322.598729] Lustre: server umount lustre-OST0001 complete [ 328.957988] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 337.514184] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 370.768340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 370.957866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 370.982710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 371.034228] Lustre: lustre-MDT0000: new disk, initializing [ 371.096868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 371.109922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 376.355108] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 386.904272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.990825] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 387.080103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 387.088025] Lustre: Skipped 1 previous similar message [ 387.156860] Lustre: lustre-MDT0001: new disk, initializing [ 387.283400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 387.343141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 387.355190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 391.633469] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 400.101577] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 404.889331] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 409.919390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 410.149378] Lustre: lustre-OST0000: new disk, initializing [ 410.154845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 410.160982] Lustre: 17068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 410.248595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.280586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 411.289718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 411.401362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 415.758801] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 425.146378] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 429.470945] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 429.735989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 434.022763] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 434.268621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 437.220312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 437.232132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 437.257268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 437.261805] Lustre: Skipped 1 previous similar message [ 442.160761] Lustre: server umount lustre-OST0000 complete [ 447.462295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 447.491432] Lustre: Skipped 1 previous similar message [ 447.507699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 447.517441] Lustre: Skipped 1 previous similar message [ 460.256144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 460.400895] Lustre: server umount lustre-MDT0000 complete [ 462.820139] LustreError: 15096:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.834718] LustreError: 15096:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 463.652912] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842940 with bad export cookie 11492017222315872579 [ 463.655078] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 463.670164] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 464.031793] Lustre: server umount lustre-MDT0001 complete [ 467.411210] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 11:02:21 (1787842941) [ 473.641306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.092869] LustreError: 19333:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.253694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.261400] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 479.714203] LustreError: 19334:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.813265] LustreError: 19333:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.518233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.063315] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 499.271450] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 505.150481] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 512.042295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 518.205191] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 525.073488] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 542.177906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 542.192435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.194074] LustreError: Skipped 1 previous similar message [ 542.197303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 542.197315] Lustre: Skipped 4 previous similar messages [ 542.227306] Lustre: Skipped 2 previous similar messages [ 546.408766] Lustre: server umount lustre-OST0000 complete [ 552.424973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.439664] Lustre: Skipped 1 previous similar message [ 556.387818] Lustre: server umount lustre-MDT0000 complete [ 557.537670] LustreError: 19333:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.549245] LustreError: 19333:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 559.785575] LustreError: 19314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843036 with bad export cookie 11492017222315873454 [ 559.790212] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 559.794678] LustreError: 19314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 560.085357] Lustre: server umount lustre-MDT0001 complete [ 569.263683] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 571.502673] Key type lgssc unregistered [ 571.772503] LNet: 22889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 571.782285] LNetError: 22889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 571.806723] LNet: Removed LNI 192.168.203.152@tcp [ 572.517165] Key type .llcrypt unregistered [ 572.519195] Key type ._llcrypt unregistered [ 591.161721] Key type ._llcrypt registered [ 591.166507] Key type .llcrypt registered [ 591.293920] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 592.459332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 592.474064] alg: No test for adler32 (adler32-zlib) [ 593.521949] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 593.794833] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 595.464172] Key type lgssc registered [ 596.480213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 603.624267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 603.637735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.016343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.669029] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 614.372138] LustreError: 24281:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.389197] LustreError: 24281:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 615.949781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.362280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 620.658321] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 627.706420] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 633.506419] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 640.223219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 640.532619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 646.726338] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 649.774168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 653.787260] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 670.184937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 670.188767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 670.205187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 670.322111] Lustre: server umount lustre-OST0000 complete [ 675.297641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.317724] Lustre: Skipped 1 previous similar message [ 675.338029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 675.350814] Lustre: Skipped 2 previous similar messages [ 679.655433] Lustre: server umount lustre-MDT0000 complete [ 680.432596] LustreError: 24281:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 680.471583] LustreError: 24281:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 683.401101] LustreError: 24261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843160 with bad export cookie 11309731053415021300 [ 683.404500] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 683.416768] LustreError: 24261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 683.832705] Lustre: server umount lustre-MDT0001 complete [ 694.442549] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 697.374713] Key type lgssc unregistered [ 697.718461] LNet: 28260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.739178] LNetError: 28260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.791972] LNet: Removed LNI 192.168.203.152@tcp [ 699.668420] Key type .llcrypt unregistered [ 699.671072] Key type ._llcrypt unregistered [ 717.422954] Key type ._llcrypt registered [ 717.429495] Key type .llcrypt registered [ 717.509891] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 718.340355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 718.390789] alg: No test for adler32 (adler32-zlib) [ 719.382937] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 719.612876] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 721.304279] Key type lgssc registered [ 722.220226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 729.764209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 729.783670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.343815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 735.923177] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 741.860558] LustreError: 29654:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 741.888235] LustreError: 29654:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 743.901204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.275397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 748.544436] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 755.738698] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 760.992083] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 772.708590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 772.988290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 777.076292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 778.309195] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 784.958364] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 797.666891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 797.673260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 797.693737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 799.715285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 799.716575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 799.716689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 803.626639] Lustre: server umount lustre-OST0000 complete [ 809.957995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 809.959413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 809.975853] Lustre: Skipped 1 previous similar message [ 809.979465] Lustre: Skipped 1 previous similar message [ 813.934207] Lustre: server umount lustre-MDT0000 complete [ 815.076936] LustreError: 29650:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 815.131030] LustreError: 29650:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 819.171604] LustreError: 29649:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 819.201457] LustreError: 29649:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 819.365807] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843296 with bad export cookie 9813137357883466986 [ 819.369791] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 819.381219] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 819.854406] Lustre: server umount lustre-MDT0001 complete [ 831.237457] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 834.710602] Key type lgssc unregistered [ 834.996570] LNet: 33554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.005863] LNetError: 33554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.026680] LNet: Removed LNI 192.168.203.152@tcp [ 835.875273] Key type .llcrypt unregistered [ 835.883403] Key type ._llcrypt unregistered [ 859.292412] Key type ._llcrypt registered [ 859.294880] Key type .llcrypt registered [ 859.393604] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 860.155929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 860.170670] alg: No test for adler32 (adler32-zlib) [ 861.160627] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 861.404593] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 863.080225] Key type lgssc registered [ 864.305734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 871.741309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 871.772313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.314256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 878.220029] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 883.682312] LustreError: 34967:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 883.709472] LustreError: 34967:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 886.980861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.383748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 892.456816] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 900.878749] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 906.357522] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 912.954720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 913.388071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.523618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 919.672409] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 926.583802] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 937.953185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 937.964964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 937.968603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 937.978520] Lustre: Skipped 1 previous similar message [ 939.957478] Lustre: server umount lustre-OST0000 complete [ 948.195493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.216575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 948.222261] Lustre: Skipped 2 previous similar messages [ 950.226264] Lustre: server umount lustre-MDT0000 complete [ 953.313178] LustreError: 34966:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 953.346247] LustreError: 34966:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 954.211625] LustreError: 34947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843431 with bad export cookie 9600702070339890312 [ 954.217721] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.231738] LustreError: 34947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 954.770813] Lustre: server umount lustre-MDT0001 complete [ 967.933715] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 971.244578] Key type lgssc unregistered [ 971.582754] LNet: 38505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.601762] LNetError: 38505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.625914] LNet: Removed LNI 192.168.203.152@tcp [ 972.862751] Key type .llcrypt unregistered [ 972.869871] Key type ._llcrypt unregistered [ 1000.288719] Key type ._llcrypt registered [ 1000.294249] Key type .llcrypt registered [ 1000.413650] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1001.298278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1001.373593] alg: No test for adler32 (adler32-zlib) [ 1002.601291] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 1002.864336] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1004.608197] Key type lgssc registered [ 1006.103497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1014.209802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1014.226311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.623692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1019.842237] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1026.029621] LustreError: 39917:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.047840] LustreError: 39917:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1028.656807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.943963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1033.359786] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1040.544931] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1045.705241] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1052.009444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1052.388189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1058.704938] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1060.598549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1066.244361] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1074.147471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1074.153815] Lustre: lustre-OST0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 1074.161341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.193945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1076.195104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.204842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1076.240112] Lustre: Skipped 2 previous similar messages [ 1078.870937] Lustre: server umount lustre-OST0000 complete [ 1100.770696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1100.779755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.790515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1105.234434] Lustre: server umount lustre-MDT0000 complete [ 1106.924198] LustreError: 40645:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.947313] LustreError: 40645:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1108.749604] LustreError: 39899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843585 with bad export cookie 2181952023674204759 [ 1108.754267] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1108.768343] LustreError: 39899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1123.296632] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1123.582056] Lustre: server umount lustre-MDT0001 complete [ 1134.990313] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1137.910027] Key type lgssc unregistered [ 1138.200804] LNet: 43629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.208428] LNetError: 43629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.227114] LNet: Removed LNI 192.168.203.152@tcp [ 1139.077342] Key type .llcrypt unregistered [ 1139.080091] Key type ._llcrypt unregistered [ 1162.325115] Key type ._llcrypt registered [ 1162.331937] Key type .llcrypt registered [ 1162.495236] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1163.402448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1163.484077] alg: No test for adler32 (adler32-zlib) [ 1164.669804] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 1164.926631] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1166.608265] Key type lgssc registered [ 1168.048067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1175.688466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1175.720658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.225112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.839659] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1187.811062] LustreError: 45041:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.844210] LustreError: 45041:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1191.270332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1191.765747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1197.238392] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1206.815416] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1212.668725] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1220.123901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1220.684095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.811452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1228.186618] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1237.059723] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1245.163812] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 1247.715995] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.718680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1247.726927] Lustre: Skipped 1 previous similar message [ 1247.737606] Lustre: Skipped 2 previous similar messages [ 1250.243141] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 1250.725423] Lustre: server umount lustre-MDT0000 complete [ 1252.835109] LustreError: 45036:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.863804] LustreError: 45036:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1254.336332] LustreError: 47625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843731 with bad export cookie 10826979976174228039 [ 1254.354378] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1254.357743] LustreError: 47625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1255.359861] LustreError: 45038:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1255.407665] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.52@tcp (stopping) [ 1257.954429] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.956447] LustreError: 45037:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.969041] Lustre: Skipped 1 previous similar message [ 1260.006052] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1260.010016] Lustre: Skipped 1 previous similar message [ 1260.704933] Lustre: server umount lustre-MDT0001 complete [ 1281.560356] Lustre: server umount lustre-OST0000 complete [ 1298.123876] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1300.666840] Key type lgssc unregistered [ 1301.043781] LNet: 48908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.048226] LNetError: 48908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.068586] LNet: Removed LNI 192.168.203.152@tcp [ 1301.908558] Key type .llcrypt unregistered [ 1301.915315] Key type ._llcrypt unregistered [ 1326.634036] Key type ._llcrypt registered [ 1326.636907] Key type .llcrypt registered [ 1326.782542] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1327.822538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1327.871175] alg: No test for adler32 (adler32-zlib) [ 1328.975520] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 1329.231305] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1330.888297] Key type lgssc registered [ 1332.315147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1340.190168] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1340.202448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1355.744526] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1360.864359] LustreError: 50296:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 1387.616486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.577623] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1403.318963] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1568.309579] Lustre: server umount lustre-OST0000 complete [ 1581.710198] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1584.090328] Key type lgssc unregistered [ 1584.319606] LNet: 51974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1584.348806] LNetError: 51974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.362633] LNet: Removed LNI 192.168.203.152@tcp [ 1584.909277] Key type .llcrypt unregistered [ 1584.918293] Key type ._llcrypt unregistered [ 1605.651154] Key type ._llcrypt registered [ 1605.656429] Key type .llcrypt registered [ 1605.768568] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1606.584057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1606.843167] alg: No test for adler32 (adler32-zlib) [ 1607.867522] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 1608.172682] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1609.832201] Key type lgssc registered [ 1610.938954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1617.555516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1617.576830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1619.056202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1622.661299] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1628.652899] LustreError: 53380:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1628.669732] LustreError: 53380:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1630.327257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.741398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1634.772951] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1641.594621] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1647.122954] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1653.034519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1653.276418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1659.212333] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1663.537432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1666.829814] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1670.627948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1670.639581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1670.666350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1673.698799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1673.701126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1673.712054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1675.458395] Lustre: server umount lustre-OST0000 complete [ 1681.896268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1681.916709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.942811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1681.945956] Lustre: Skipped 1 previous similar message [ 1683.940252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1683.947320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1683.965082] Lustre: Skipped 1 previous similar message [ 1689.059868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1689.077871] Lustre: Skipped 1 previous similar message [ 1694.176153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1694.566508] Lustre: server umount lustre-MDT0000 complete [ 1698.277414] LustreError: 53381:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1698.298389] LustreError: 53381:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1698.834155] LustreError: 54116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844175 with bad export cookie 14952024951532789708 [ 1698.835899] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1698.852792] LustreError: 54116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1699.554437] Lustre: server umount lustre-MDT0001 complete [ 1710.758523] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1714.126491] Key type lgssc unregistered [ 1714.516770] LNet: 56915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.532135] LNetError: 56915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1714.552746] LNet: Removed LNI 192.168.203.152@tcp [ 1715.626494] Key type .llcrypt unregistered [ 1715.630655] Key type ._llcrypt unregistered [ 1740.782606] Key type ._llcrypt registered [ 1740.788241] Key type .llcrypt registered [ 1740.947576] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1742.176530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1742.216296] alg: No test for adler32 (adler32-zlib) [ 1743.347475] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 1743.635697] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1745.424920] Key type lgssc registered [ 1746.721912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1754.353673] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1754.401338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1770.083191] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1775.264572] LustreError: 58301:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 1801.953368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.641271] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1816.077112] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1822.787898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.977978] LustreError: 57636:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ab4c1f07 x1874690540833024/t0(0) o250->MGC192.168.203.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1823.443781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.572988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1828.056842] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1828.837703] LustreError: 58327:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1828.869779] LustreError: 58327:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1833.968474] LustreError: 58326:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1836.105207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1836.461378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1840.500447] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1847.297888] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1852.965586] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1856.999753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1857.009729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1857.027767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1858.022913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1858.024305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.141382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1863.151787] Lustre: Skipped 1 previous similar message [ 1868.267116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1868.279840] Lustre: Skipped 1 previous similar message [ 1869.281435] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1869.474959] Lustre: server umount lustre-OST0000 complete [ 1890.224616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.242045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1890.249518] Lustre: Skipped 1 previous similar message [ 1896.527776] Lustre: server umount lustre-MDT0000 complete [ 1898.466858] LustreError: 59340:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1898.496462] LustreError: 59340:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1899.933038] LustreError: 58309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844376 with bad export cookie 7174867524918265717 [ 1899.939684] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1899.944991] LustreError: 58309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1914.336116] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1914.659993] Lustre: server umount lustre-MDT0001 complete [ 1926.992653] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1930.935989] Key type lgssc unregistered [ 1931.297583] LNet: 62050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.307413] LNetError: 62050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.324819] LNet: Removed LNI 192.168.203.152@tcp [ 1932.359244] Key type .llcrypt unregistered [ 1932.361549] Key type ._llcrypt unregistered [ 1958.128615] Key type ._llcrypt registered [ 1958.131418] Key type .llcrypt registered [ 1958.291357] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1959.813298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1959.835793] alg: No test for adler32 (adler32-zlib) [ 1960.960800] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 1961.351445] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1963.064114] Key type lgssc registered [ 1964.612990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1972.408939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1972.426257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.904583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.554338] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1983.463390] LustreError: 63461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.484136] LustreError: 63461:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1986.413597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.769313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1990.976837] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1998.034764] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2003.507300] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2011.159409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2011.560729] LustreError: 65284:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2011.782712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.024629] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2018.312572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2025.108431] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2058.214095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2058.236182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.280408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2059.233508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2059.238107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2059.266294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.714147] Lustre: server umount lustre-OST0000 complete [ 2068.964292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2068.978437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2068.989889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2068.998966] Lustre: Skipped 1 previous similar message [ 2071.994512] Lustre: server umount lustre-MDT0000 complete [ 2074.598929] LustreError: 63458:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.637931] LustreError: 63458:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2076.630409] LustreError: 63442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844553 with bad export cookie 14459001580010748437 [ 2076.639940] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2076.660800] LustreError: 63442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2077.177629] Lustre: server umount lustre-MDT0001 complete [ 2087.844208] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2090.811777] Key type lgssc unregistered [ 2091.095771] LNet: 67025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.104601] LNetError: 67025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2091.128518] LNet: Removed LNI 192.168.203.152@tcp [ 2091.972769] Key type .llcrypt unregistered [ 2091.976655] Key type ._llcrypt unregistered [ 2120.203636] Key type ._llcrypt registered [ 2120.214421] Key type .llcrypt registered [ 2120.397662] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2122.058306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2122.110646] alg: No test for adler32 (adler32-zlib) [ 2123.244951] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 2123.518502] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2125.208941] Key type lgssc registered [ 2126.516317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2136.696570] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 11:30:10 (1787844610) [ 2144.038246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2144.070677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.599035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.641103] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2156.005444] LustreError: 68733:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2156.029334] LustreError: 68733:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2157.886611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.287526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2162.999248] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2169.925892] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2175.109111] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2181.139098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2181.473926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.742875] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2190.731437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2194.875811] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2200.994740] Lustre: Failing over lustre-MDT0000 [ 2201.066027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2201.083244] Lustre: Skipped 2 previous similar messages [ 2201.089292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2201.094192] Lustre: Skipped 2 previous similar messages [ 2201.301971] Lustre: server umount lustre-MDT0000 complete [ 2202.615268] LustreError: 70983:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.186584] LustreError: 68733:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.197710] LustreError: 68733:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2210.480184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.614135] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2210.775777] Lustre: *** cfs_fail_loc=135, val=0*** [ 2210.913962] LustreError: 71819:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2210.922712] LustreError: 71819:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.152@tcp: cfg command failed: rc = -2 [ 2210.933194] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2210.933194] [ 2210.944870] LustreError: MGC192.168.203.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2210.971354] LustreError: 71808:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2210.992099] LustreError: 71808:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2210.998586] LustreError: 71808:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2211.007646] LustreError: 71808:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2211.046289] Lustre: server umount lustre-MDT0000 complete [ 2211.048519] LustreError: 71808:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2211.302873] LustreError: 68732:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.327880] LustreError: 68732:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2216.417860] LustreError: 69460:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.436908] LustreError: 69460:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2217.585077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.702616] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2217.937384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.993750] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2222.446876] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2223.051864] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2223.096873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2223.129102] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2223.213619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2233.319978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.333495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2233.343609] Lustre: Skipped 1 previous similar message [ 2236.940819] Lustre: server umount lustre-OST0000 complete [ 2243.557710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.558821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2243.576716] Lustre: Skipped 1 previous similar message [ 2243.607318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2243.621444] Lustre: Skipped 1 previous similar message [ 2246.906603] Lustre: server umount lustre-MDT0000 complete [ 2248.678694] LustreError: 68732:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.692971] LustreError: 68732:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2250.585362] LustreError: 68713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844727 with bad export cookie 1936751195110404719 [ 2250.594917] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2250.610720] LustreError: 68713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2251.080692] Lustre: server umount lustre-MDT0001 complete [ 2263.243741] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2266.423412] Key type lgssc unregistered [ 2266.632663] LNet: 73601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2266.641650] LNetError: 73601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2266.702318] LNet: Removed LNI 192.168.203.152@tcp [ 2267.696070] Key type .llcrypt unregistered [ 2267.699464] Key type ._llcrypt unregistered [ 2289.973655] Key type ._llcrypt registered [ 2289.978486] Key type .llcrypt registered [ 2290.118575] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2291.242443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2291.339605] alg: No test for adler32 (adler32-zlib) [ 2292.429474] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 2292.790587] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2294.458082] Key type lgssc registered [ 2295.428879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2302.798115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2302.814344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2304.310225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.516805] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2314.730241] LustreError: 75016:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.751402] LustreError: 75016:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2316.367466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.771122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2321.112870] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2328.071859] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2334.042503] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2341.862417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2342.222158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.432855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2350.036987] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2359.185196] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2365.158103] Lustre: Failing over lustre-MDT0000 [ 2365.451875] Lustre: server umount lustre-MDT0000 complete [ 2366.948750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.968068] Lustre: Skipped 1 previous similar message [ 2366.975888] LustreError: 75743:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2367.022142] LustreError: 75743:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2372.041732] LustreError: 75013:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.006665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.215507] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2374.348425] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2374.502995] LustreError: 78103:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2374.512791] LustreError: 78103:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.152@tcp: cfg command failed: rc = -9 [ 2374.530439] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2374.530439] [ 2374.548875] LustreError: MGC192.168.203.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2374.571732] LustreError: 78091:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2374.580790] LustreError: 78091:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2374.591484] LustreError: 78091:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2374.606741] LustreError: 78091:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2374.650728] Lustre: server umount lustre-MDT0000 complete [ 2374.655142] LustreError: 78091:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2377.162107] LustreError: 75011:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.196540] LustreError: 75011:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2382.279873] LustreError: 75012:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.935786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.121292] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.461824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.532267] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2387.410933] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2388.501501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2388.531875] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2388.609255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2389.553839] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2397.666036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2397.674450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.689601] Lustre: Skipped 1 previous similar message [ 2397.694992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2398.694666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.695650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2398.715924] Lustre: Skipped 1 previous similar message [ 2403.609332] Lustre: server umount lustre-OST0000 complete [ 2408.929706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2408.930021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2408.935556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2408.964104] Lustre: Skipped 1 previous similar message [ 2414.050147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2414.061666] Lustre: Skipped 2 previous similar messages [ 2419.170147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.178314] Lustre: Skipped 1 previous similar message [ 2421.216294] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2421.522463] Lustre: server umount lustre-MDT0000 complete [ 2423.266422] LustreError: 75016:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.276133] LustreError: 75016:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2424.742978] LustreError: 74997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844901 with bad export cookie 8814774758171512630 [ 2424.745940] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2424.753440] LustreError: 74997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2425.110382] Lustre: server umount lustre-MDT0001 complete [ 2436.012816] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2438.843587] Key type lgssc unregistered [ 2439.117724] LNet: 79890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.126168] LNetError: 79890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2439.140863] LNet: Removed LNI 192.168.203.152@tcp [ 2439.977562] Key type .llcrypt unregistered [ 2439.983389] Key type ._llcrypt unregistered [ 2454.015266] Key type ._llcrypt registered [ 2454.019341] Key type .llcrypt registered [ 2454.117621] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2454.429819] Key type .llcrypt unregistered [ 2454.432177] Key type ._llcrypt unregistered [ 2477.098424] Key type ._llcrypt registered [ 2477.101200] Key type .llcrypt registered [ 2477.200169] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2478.377045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2478.399077] alg: No test for adler32 (adler32-zlib) [ 2479.530393] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 2479.854045] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2481.568164] Key type lgssc registered [ 2483.004220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2490.352962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2490.370989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.028816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2496.544850] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2501.624964] LustreError: 81810:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.659394] LustreError: 81810:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2505.755026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.276924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2511.345419] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2518.879203] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2524.596735] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2531.572651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2532.057859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.143692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2538.585541] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2546.264896] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2560.996752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2561.016525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2561.040781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2565.835911] Lustre: server umount lustre-OST0000 complete [ 2571.235834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.239047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2571.255517] Lustre: Skipped 2 previous similar messages [ 2571.274732] Lustre: Skipped 2 previous similar messages [ 2576.089798] Lustre: server umount lustre-MDT0000 complete [ 2576.353689] LustreError: 82790:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.370305] LustreError: 82790:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2579.061677] LustreError: 81790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845055 with bad export cookie 15287222628136435121 [ 2579.069491] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2579.071036] LustreError: 81790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2579.308034] Lustre: server umount lustre-MDT0001 complete [ 2588.157324] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2590.950174] Key type lgssc unregistered [ 2591.360247] LNet: 85371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2591.368575] LNetError: 85371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2591.386634] LNet: Removed LNI 192.168.203.152@tcp [ 2592.621644] Key type .llcrypt unregistered [ 2592.625065] Key type ._llcrypt unregistered [ 2616.905764] Key type ._llcrypt registered [ 2616.908689] Key type .llcrypt registered [ 2617.016219] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2617.837453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2617.887244] alg: No test for adler32 (adler32-zlib) [ 2618.950794] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 2619.131890] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2620.824240] Key type lgssc registered [ 2621.982136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2628.677230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2628.693781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2630.104578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.016816] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2639.337059] LustreError: 86786:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.354839] LustreError: 86786:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2641.436144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.924290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2646.472698] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2653.722658] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2659.212438] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2666.767496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2667.300274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2673.832373] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2674.503919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2681.476347] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2692.068853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2692.070918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2692.088473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2692.105628] Lustre: Skipped 1 previous similar message [ 2693.948748] Lustre: server umount lustre-OST0000 complete [ 2697.192909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.195180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2697.207269] Lustre: Skipped 1 previous similar message [ 2697.219493] Lustre: Skipped 2 previous similar messages [ 2702.305283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2702.308134] Lustre: Skipped 1 previous similar message [ 2703.232430] Lustre: server umount lustre-MDT0000 complete [ 2706.186185] LustreError: 86766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845183 with bad export cookie 7723702902348545723 [ 2706.188669] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2706.198164] LustreError: 86766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2706.506473] Lustre: server umount lustre-MDT0001 complete [ 2716.024772] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2718.977398] Key type lgssc unregistered [ 2719.244761] LNet: 90317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2719.257576] LNetError: 90317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2719.289960] LNet: Removed LNI 192.168.203.152@tcp [ 2720.212264] Key type .llcrypt unregistered [ 2720.217131] Key type ._llcrypt unregistered [ 2751.681484] Key type ._llcrypt registered [ 2751.684141] Key type .llcrypt registered [ 2751.982378] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2753.528498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2753.567039] alg: No test for adler32 (adler32-zlib) [ 2754.630698] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 2754.883778] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2756.649211] Key type lgssc registered [ 2757.999899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2765.680030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2765.702057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.259366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2771.611119] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2777.573578] LustreError: 91731:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.607800] LustreError: 91731:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2780.230084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2780.561938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2784.911243] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2791.426583] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2796.336475] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2802.341520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2802.769826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2808.872058] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2813.057212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2815.409690] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2826.740585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2826.740702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2826.749737] LustreError: Skipped 1 previous similar message [ 2826.759345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2826.774914] Lustre: Skipped 1 previous similar message [ 2826.810590] Lustre: Skipped 1 previous similar message [ 2828.290920] Lustre: server umount lustre-OST0000 complete [ 2831.841425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2831.855764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.883362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2833.889052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2833.891535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2838.039673] Lustre: server umount lustre-MDT0000 complete [ 2839.012562] LustreError: 92459:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2839.026769] LustreError: 92459:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2841.587265] LustreError: 91713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845318 with bad export cookie 15338934138247240384 [ 2841.595961] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2841.597747] LustreError: 91713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2841.852701] Lustre: server umount lustre-MDT0001 complete [ 2854.529619] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2857.807860] Key type lgssc unregistered [ 2858.385830] LNet: 95267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2858.399016] LNetError: 95267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2858.436747] LNet: Removed LNI 192.168.203.152@tcp [ 2859.472418] Key type .llcrypt unregistered [ 2859.474133] Key type ._llcrypt unregistered [ 2884.988106] Key type ._llcrypt registered [ 2884.992845] Key type .llcrypt registered [ 2885.133512] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2886.144565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2886.299924] alg: No test for adler32 (adler32-zlib) [ 2887.394834] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 2887.699792] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2889.368761] Key type lgssc registered [ 2890.590200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2897.762780] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2897.798771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2913.376561] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2918.496295] LustreError: 96653:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 2945.122826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.921355] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2960.417385] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2965.416121] Lustre: server umount lustre-OST0000 complete [ 2973.246478] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 11:44:07 (1787845447) [ 2979.845467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2980.352609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2985.441860] LustreError: 98293:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.473974] LustreError: 98293:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2985.852920] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2990.567683] LustreError: 98294:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.685460] LustreError: 98293:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.766571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2996.123387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3000.367425] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3007.641403] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3012.501909] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3018.946707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3019.392470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3024.926593] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3025.555740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3033.449809] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3046.370659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3046.384474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3046.387634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3046.403610] Lustre: Skipped 1 previous similar message [ 3051.492519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3051.501034] Lustre: Skipped 1 previous similar message [ 3051.849863] Lustre: server umount lustre-OST0000 complete [ 3056.613229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3056.616763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3056.627468] Lustre: Skipped 1 previous similar message [ 3056.632372] Lustre: Skipped 1 previous similar message [ 3061.530271] Lustre: server umount lustre-MDT0000 complete [ 3061.733273] LustreError: 99330:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3061.750248] LustreError: 99330:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3065.172729] LustreError: 98273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845542 with bad export cookie 6119446354282864397 [ 3065.188952] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3065.194830] LustreError: 98273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3065.771884] Lustre: server umount lustre-MDT0001 complete [ 3076.007826] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 3078.764777] Key type lgssc unregistered [ 3079.116733] LNet: 101988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3079.122093] LNetError: 101988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3079.133610] LNet: Removed LNI 192.168.203.152@tcp [ 3080.096264] Key type .llcrypt unregistered [ 3080.100320] Key type ._llcrypt unregistered [ 3121.805310] Key type ._llcrypt registered [ 3121.812453] Key type .llcrypt registered [ 3121.943927] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3122.865604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3122.898222] alg: No test for adler32 (adler32-zlib) [ 3124.001523] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 3124.290448] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 3125.952156] Key type lgssc registered [ 3127.094495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3135.149536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3135.177577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.679731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3141.560955] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3146.221164] LustreError: 104173:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.261833] LustreError: 104173:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3151.098976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3151.334654] LustreError: 104174:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3151.723896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3157.846927] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3165.934883] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3172.407369] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3179.882902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3180.258204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3181.308293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3186.910969] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3194.534314] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3207.138787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3207.140477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3207.179905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3207.197155] Lustre: Skipped 1 previous similar message [ 3210.129614] Lustre: server umount lustre-OST0000 complete [ 3217.378389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.385987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3217.403075] Lustre: Skipped 2 previous similar messages [ 3220.327406] Lustre: server umount lustre-MDT0000 complete [ 3222.507927] LustreError: 104174:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3222.555149] LustreError: 104174:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3224.765333] LustreError: 104152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845701 with bad export cookie 1501248102622557831 [ 3224.776662] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3224.784254] LustreError: 104152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3225.354941] Lustre: server umount lustre-MDT0001 complete [ 3236.360853] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 3239.576437] Key type lgssc unregistered [ 3239.960531] LNet: 107710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3239.972577] LNetError: 107710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3240.012077] LNet: Removed LNI 192.168.203.152@tcp [ 3240.889119] Key type .llcrypt unregistered [ 3240.891613] Key type ._llcrypt unregistered [ 3260.091757] Key type ._llcrypt registered [ 3260.094076] Key type .llcrypt registered [ 3260.224413] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3261.613840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3261.715057] alg: No test for adler32 (adler32-zlib) [ 3262.872350] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 3263.155522] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 3264.840345] Key type lgssc registered [ 3266.500985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3276.136760] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3276.172880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3291.872488] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3296.992295] LustreError: 108952:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 3323.683754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3329.218534] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3336.505755] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3342.939765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.191263] LustreError: MGC192.168.203.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3343.200645] LustreError: 109985:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3343.205817] LustreError: 109985:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3343.211792] LustreError: 109985:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3343.282475] Lustre: server umount lustre-MDT0000 complete [ 3343.286451] LustreError: 109985:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3364.322866] Lustre: 108965:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845825/real 1787845825] req@0000000061a97767 x1874692133621120/t0(0) o101->MGC192.168.203.152@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1787845841 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3364.372279] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3364.393449] LustreError: 108965:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3364.582643] Lustre: server umount lustre-OST0000 complete [ 3374.569650] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 3385.265818] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3426.356837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.621173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3426.654901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3426.747284] Lustre: lustre-MDT0000: new disk, initializing [ 3426.879965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3426.901329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3431.679939] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3441.249467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3441.361455] Lustre: 112779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3441.430779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3441.436469] Lustre: Skipped 1 previous similar message [ 3441.505173] Lustre: lustre-MDT0001: new disk, initializing [ 3441.611951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3441.657341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3441.672689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3445.645182] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3454.173984] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3459.217986] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3465.422919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3465.771185] Lustre: lustre-OST0000: new disk, initializing [ 3465.779227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3465.792306] Lustre: 114750:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3465.901372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3471.430180] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3472.459837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3472.475985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3472.581324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3480.156777] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3484.869589] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3485.141659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3490.078774] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3490.338893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3493.364134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3493.373457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3493.387673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3493.397553] Lustre: Skipped 1 previous similar message [ 3498.499194] Lustre: server umount lustre-OST0000 complete [ 3503.073620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3503.081122] LustreError: Skipped 1 previous similar message [ 3503.097513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.117407] Lustre: Skipped 1 previous similar message [ 3503.139272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3508.629668] Lustre: server umount lustre-MDT0000 complete [ 3512.032109] LustreError: 112771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845988 with bad export cookie 16169200966376113200 [ 3512.037070] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3512.049379] LustreError: 112771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3512.396042] Lustre: server umount lustre-MDT0001 complete [ 3521.068621] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 11:53:15 (1787845995) [ 3524.326179] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3541.144400] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 3552.102818] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3583.776207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3584.073935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3584.100882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3584.168728] Lustre: lustre-MDT0000: new disk, initializing [ 3584.241722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3584.256380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3588.954819] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3598.741216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3598.930315] Lustre: 119650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3598.978896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3598.985786] Lustre: Skipped 1 previous similar message [ 3599.038603] Lustre: lustre-MDT0001: new disk, initializing [ 3599.167852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3599.202429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3599.214791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3603.720832] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3613.512030] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3620.512442] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3629.815426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3630.184934] Lustre: lustre-OST0000: new disk, initializing [ 3630.194554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3630.209986] Lustre: 121619:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3630.359359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3635.780455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3635.810181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3635.934049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3636.818469] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3647.728772] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3653.638970] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3654.045505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3659.924941] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3660.220581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3666.913177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3666.923591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3666.928606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3666.945027] Lustre: Skipped 2 previous similar messages [ 3666.955658] Lustre: Skipped 2 previous similar messages [ 3668.609215] Lustre: server umount lustre-OST0000 complete [ 3676.136707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3676.154723] LustreError: Skipped 1 previous similar message [ 3676.158776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3676.175138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3678.585759] Lustre: server umount lustre-MDT0000 complete [ 3682.220786] LustreError: 119642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846159 with bad export cookie 16169200966376114082 [ 3682.256171] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3682.686601] Lustre: server umount lustre-MDT0001 complete [ 3690.761248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3691.189602] LustreError: 123835:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.210630] LustreError: 123835:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3691.315263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3695.553991] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3696.616607] LustreError: 123836:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3701.730280] LustreError: 123835:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3703.237906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.764901] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3714.822329] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3719.583230] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3725.663243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3731.678686] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3739.248907] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3745.912177] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3752.857987] Lustre: server umount lustre-OST0000 complete [ 3758.059686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.073633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3758.076683] Lustre: Skipped 2 previous similar messages [ 3758.087227] Lustre: Skipped 2 previous similar messages [ 3766.405398] Lustre: server umount lustre-MDT0000 complete [ 3768.289440] Lustre: 108289:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846229/real 1787846229] req@00000000273f3718 x1874692133729920/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787846245 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3768.295193] LustreError: 123835:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.339576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3768.388906] LustreError: 123835:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3770.339439] Lustre: 108291:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846231/real 1787846231] req@0000000064ba238b x1874692133730176/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787846247 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3770.612115] LustreError: 125680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846247 with bad export cookie 16169200966376114957 [ 3770.620974] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3770.632768] LustreError: 125680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3770.981146] Lustre: server umount lustre-MDT0001 complete [ 3782.010953] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 3784.821900] Key type lgssc unregistered [ 3785.163928] LNet: 127466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3785.172536] LNetError: 127466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3785.183040] LNet: Removed LNI 192.168.203.152@tcp [ 3786.136681] Key type .llcrypt unregistered [ 3786.139109] Key type ._llcrypt unregistered [ 3803.140315] Key type ._llcrypt registered [ 3803.142731] Key type .llcrypt registered [ 3803.263463] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 3818.770854] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3820.427236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3820.474209] alg: No test for adler32 (adler32-zlib) [ 3821.672707] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 3821.988856] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 3823.720150] Key type lgssc registered [ 3824.843363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3860.382665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3860.414991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3861.638645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3861.682278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3861.854129] Lustre: lustre-MDT0000: new disk, initializing [ 3861.989831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.006735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3866.802646] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3876.870410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3877.057298] Lustre: 130620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3877.155499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3877.166579] Lustre: Skipped 1 previous similar message [ 3877.252936] Lustre: lustre-MDT0001: new disk, initializing [ 3877.387965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3877.441684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3877.457996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3883.103336] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3892.328680] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3898.080597] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3904.461521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3904.833221] Lustre: lustre-OST0000: new disk, initializing [ 3904.843147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3904.858584] Lustre: 132586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3904.962676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.700956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3910.721922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3910.796610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3911.122780] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3920.592386] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3925.307930] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3925.465923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3929.895314] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3930.142281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3936.231385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3936.245314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3936.260297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3936.737448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3936.746938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3936.775227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3938.034854] Lustre: server umount lustre-OST0000 complete [ 3941.858348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.863991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3941.873993] Lustre: Skipped 1 previous similar message [ 3941.888993] Lustre: Skipped 1 previous similar message [ 3946.980497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3946.996517] Lustre: Skipped 1 previous similar message [ 3948.070657] Lustre: server umount lustre-MDT0000 complete [ 3951.598153] LustreError: 130611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846428 with bad export cookie 17878075046250617561 [ 3951.611109] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3951.617908] LustreError: 130611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3951.995812] Lustre: server umount lustre-MDT0001 complete [ 3960.117400] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 12:00:34 (1787846434) [ 3966.487995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3967.070520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3971.890501] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3972.066310] LustreError: 134999:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3972.092927] LustreError: 134999:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3977.188811] LustreError: 134998:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.098514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3980.465285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3984.610731] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3992.382475] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3998.488668] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4000.553751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.585619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4000.596198] Lustre: Skipped 1 previous similar message [ 4006.870982] Lustre: server umount lustre-MDT0000 complete [ 4010.522133] LustreError: 136578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846487 with bad export cookie 17878075046250618443 [ 4010.527567] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4010.534938] LustreError: 136578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4010.895973] Lustre: server umount lustre-MDT0001 complete [ 4019.185469] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 12:01:33 (1787846493) [ 4026.063040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4041.696588] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4046.817074] LustreError: 137425:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 4072.611116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4080.611578] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4088.667915] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4090.827208] Lustre: server umount lustre-OST0000 complete [ 4100.544321] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 12:02:54 (1787846574) [ 4109.429478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4110.254157] LustreError: 138871:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4110.314468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4115.296591] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4115.425873] LustreError: 138872:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.560801] LustreError: 138871:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4125.870796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4131.984918] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4139.787427] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4145.656311] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4152.671417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4153.115719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4153.129429] Lustre: Skipped 1 previous similar message [ 4159.975801] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4167.709712] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4180.961527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4180.977566] Lustre: Skipped 1 previous similar message [ 4180.984803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4180.996861] Lustre: Skipped 4 previous similar messages [ 4184.152386] Lustre: server umount lustre-MDT0000 complete [ 4186.085302] LustreError: 138872:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4186.122436] LustreError: 138872:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4188.036483] LustreError: 138851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846664 with bad export cookie 17878075046250619108 [ 4188.044313] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4188.044898] LustreError: 138851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4188.441889] Lustre: server umount lustre-MDT0001 complete [ 4201.828870] Lustre: server umount lustre-OST0000 complete [ 4210.113968] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 12:04:44 (1787846684) [ 4216.086847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4216.687914] LustreError: 142528:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.430934] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4231.176146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4231.631838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4231.643697] Lustre: Skipped 1 previous similar message [ 4236.683380] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4244.173332] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4250.714593] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4257.889297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4261.446406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4265.349250] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4273.161931] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4278.821833] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4279.090547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4285.117312] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4285.392891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4292.074223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4292.077197] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4292.078358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4292.094361] Lustre: Skipped 2 previous similar messages [ 4293.737423] Lustre: server umount lustre-OST0000 complete [ 4298.209439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4303.558561] Lustre: server umount lustre-MDT0000 complete [ 4307.368216] LustreError: 143264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846784 with bad export cookie 17878075046250620179 [ 4307.374541] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4307.382139] LustreError: 143264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4307.743234] Lustre: server umount lustre-MDT0001 complete [ 4316.449483] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 12:06:30 (1787846790) [ 4323.290924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4339.040897] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4344.161056] LustreError: 146687:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 4369.888352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4369.901613] Lustre: Skipped 1 previous similar message [ 4377.431575] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4386.139252] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4393.797340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4394.257549] LustreError: 146712:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.292840] LustreError: 146712:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4394.440262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4400.957545] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4410.954937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4418.278634] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4428.508942] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4435.540398] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4440.378477] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4440.808283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4445.956556] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4446.181174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4452.327617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4452.329235] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4452.331056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4452.331062] Lustre: Skipped 4 previous similar messages [ 4452.368302] Lustre: Skipped 4 previous similar messages [ 4454.213991] Lustre: server umount lustre-MDT0000 complete [ 4457.997763] LustreError: 146694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846934 with bad export cookie 17878075046250621201 [ 4458.007470] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4458.009750] LustreError: 146694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4458.478649] Lustre: server umount lustre-MDT0001 complete [ 4472.983872] Lustre: server umount lustre-OST0000 complete [ 4482.415795] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 12:09:16 (1787846956) [ 4489.054042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4504.610232] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4509.729240] LustreError: 150886:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 4543.106258] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4552.184333] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4562.390795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4563.034425] LustreError: 150911:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4563.081365] LustreError: 150911:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 4563.258199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4569.360175] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4579.248059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4585.195701] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4593.911161] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4601.818339] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4609.712044] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4609.880624] Lustre: lustre-OST0001: new disk, initializing [ 4609.884434] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4609.891556] Lustre: 153762:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4618.161477] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4619.832166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4619.843412] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4620.037931] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4627.838458] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4632.533490] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4632.764824] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4637.142622] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4637.402618] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4640.226346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4640.237084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4640.257381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4640.262594] Lustre: Skipped 2 previous similar messages [ 4645.451525] Lustre: server umount lustre-OST0000 complete [ 4663.264164] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4663.567567] Lustre: server umount lustre-OST0001 complete [ 4673.307899] Lustre: server umount lustre-MDT0000 complete [ 4677.066716] LustreError: 150893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847153 with bad export cookie 17878075046250622153 [ 4677.078653] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4677.088039] LustreError: 150893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4677.414499] Lustre: server umount lustre-MDT0001 complete [ 4703.976546] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 12:12:58 (1787847178) [ 4706.416296] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4708.342408] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4710.221814] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 12:13:04 (1787847184) [ 4717.217697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4717.690053] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4717.726149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4717.940321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4717.953514] Lustre: Skipped 6 previous similar messages [ 4722.898404] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4732.498891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4732.606658] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4732.653491] Lustre: 157321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4737.302932] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4745.443324] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4751.155342] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4758.103907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4758.329704] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4765.126182] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4766.434658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4774.351670] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4779.101885] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4779.318023] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4784.010962] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4784.212694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4786.669900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4786.684365] LustreError: Skipped 2 previous similar messages [ 4786.690727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4786.705765] Lustre: Skipped 5 previous similar messages [ 4786.717149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4786.724798] Lustre: Skipped 9 previous similar messages [ 4792.515926] Lustre: server umount lustre-OST0000 complete [ 4814.935655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4817.456415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4817.487586] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4822.197789] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4822.637046] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4822.667781] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4830.810619] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4838.069974] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4838.607306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4844.601429] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4844.885933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4851.478354] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8fe745cda800.ost_server_uuid 50 [ 4852.864974] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8fe745cda800.ost_server_uuid in FULL state after 0 sec [ 4853.125234] LustreError: 157329:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.52@tcp) returned error from blocking AST (req@000000000ab6ea24 x1874692720038656 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000015ec99a2/0xf81bae884f8b3623 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.52@tcp remote: 0xcff46044f810303b expref: 5 pid: 157329 timeout: 4953 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4853.172304] LustreError: lustre-MDT0000: A client on nid 192.168.203.52@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4853.182677] LustreError: 157319:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.52@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000015ec99a2/0xf81bae884f8b3623 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.52@tcp remote: 0xcff46044f810303b expref: 6 pid: 157329 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4863.976273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4863.993545] LustreError: Skipped 1 previous similar message [ 4873.185935] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4873.476855] Lustre: server umount lustre-OST0000 complete [ 4884.136564] Lustre: server umount lustre-MDT0000 complete [ 4884.451797] LustreError: 157329:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4884.482785] LustreError: 157329:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 16 previous similar messages [ 4888.124319] LustreError: 159185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847364 with bad export cookie 17878075046250623336 [ 4888.128191] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4888.140993] LustreError: 159185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4894.647941] Lustre: server umount lustre-MDT0001 complete [ 4908.126325] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 4911.282708] Key type lgssc unregistered [ 4911.578054] LNet: 163442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4911.601794] LNetError: 163442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4911.639339] LNet: Removed LNI 192.168.203.152@tcp [ 4912.574296] Key type .llcrypt unregistered [ 4912.576849] Key type ._llcrypt unregistered [ 4939.416526] Key type ._llcrypt registered [ 4939.425556] Key type .llcrypt registered [ 4939.615968] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 4941.024341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4941.295158] alg: No test for adler32 (adler32-zlib) [ 4942.534512] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 4942.846548] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 4944.522753] Key type lgssc registered [ 4947.106711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4957.647391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4957.686599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4959.434692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4966.295582] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4969.968940] LustreError: 164856:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4970.005623] LustreError: 164856:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4975.083442] LustreError: 164855:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.792235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4977.070676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4983.137378] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4991.577237] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4997.552740] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5004.673599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5004.790059] LustreError: 166680:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5005.079989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5010.419873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5011.296948] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5018.979885] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5028.726809] Lustre: Failing over lustre-MDT0000 [ 5029.104668] Lustre: server umount lustre-MDT0000 complete [ 5031.907425] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5031.910606] LustreError: 164855:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5031.927618] Lustre: Skipped 1 previous similar message [ 5031.951351] LustreError: 164855:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5037.026864] LustreError: 165785:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5037.063289] LustreError: 165785:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5041.946868] LustreError: 164835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.52@tcp arrived at 1787847518 with bad export cookie 9290602761697038997 [ 5041.968021] LustreError: 164835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5047.266056] Lustre: 164168:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787847508/real 1787847508] req@00000000210d9a51 x1874693895255168/t0(0) o400->MGC192.168.203.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787847524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5047.274187] LustreError: 164850:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5047.307906] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5047.372262] LustreError: 164850:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 5050.392474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.507605] LustreError: 164165:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a2423115 x1874693895260544/t0(0) o250->MGC192.168.203.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5057.917330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5057.991880] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5062.993524] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5063.139075] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5063.162570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5065.335615] Lustre: lustre-MDT0000: Denying connection for new client cc44c9f5-67a3-4090-9345-0499828ce73f (at 192.168.203.52@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5077.989714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5077.996808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5078.014358] Lustre: Skipped 1 previous similar message [ 5078.030085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5079.760564] Lustre: server umount lustre-OST0000 complete [ 5083.707196] LustreError: 168587:0:(ldlm_lib.c:3067:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5083.712265] Lustre: 167934:0:(ldlm_lib.c:2467:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5083.717239] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5083.731331] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5083.737515] LustreError: 167934:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d9c3f45b x1874693895275392/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5083.751387] LustreError: 167934:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5083.758076] LustreError: 167934:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5083.765885] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5084.029564] Lustre: server umount lustre-MDT0000 complete [ 5087.795740] LustreError: 164835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847564 with bad export cookie 9290602761697039102 [ 5087.804054] LustreError: 164165:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5087.804693] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5087.821278] LustreError: 164835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5090.172969] Lustre: server umount lustre-MDT0001 complete [ 5101.114133] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5104.081285] Key type lgssc unregistered [ 5104.449766] LNet: 169346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5104.458222] LNetError: 169346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5104.481416] LNet: Removed LNI 192.168.203.152@tcp [ 5105.512529] Key type .llcrypt unregistered [ 5105.515659] Key type ._llcrypt unregistered [ 5130.996464] Key type ._llcrypt registered [ 5131.003779] Key type .llcrypt registered [ 5131.119189] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5132.256159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5132.339900] alg: No test for adler32 (adler32-zlib) [ 5133.413346] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 5133.706411] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5135.464196] Key type lgssc registered [ 5136.775114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5144.570518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5144.609845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5146.101480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5151.091077] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5155.305382] LustreError: 170757:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5155.319451] LustreError: 170757:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5160.420915] LustreError: 170758:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5160.793375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.213399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5165.774899] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5172.945380] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5178.914773] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5185.944761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5186.398879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5190.516412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5193.680523] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5202.298964] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5211.106831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5211.109125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5211.109715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5211.123064] Lustre: Skipped 1 previous similar message [ 5214.308156] Lustre: server umount lustre-OST0000 complete [ 5221.347293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5221.352254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5221.358367] Lustre: Skipped 1 previous similar message [ 5221.362933] Lustre: Skipped 1 previous similar message [ 5225.157986] Lustre: server umount lustre-MDT0000 complete [ 5226.470133] LustreError: 170758:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5226.513469] LustreError: 170758:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5229.292135] LustreError: 173538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847706 with bad export cookie 14218103233664834328 [ 5229.294509] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5229.314456] LustreError: 173538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5229.742310] Lustre: server umount lustre-MDT0001 complete [ 5240.205042] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5242.881786] Key type lgssc unregistered [ 5243.139593] LNet: 174293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5243.146442] LNetError: 174293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5243.182766] LNet: Removed LNI 192.168.203.152@tcp [ 5244.170054] Key type .llcrypt unregistered [ 5244.173687] Key type ._llcrypt unregistered [ 5277.091431] Key type ._llcrypt registered [ 5277.094762] Key type .llcrypt registered [ 5277.188426] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5278.639773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5278.816306] alg: No test for adler32 (adler32-zlib) [ 5279.953981] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 5280.188982] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5281.857276] Key type lgssc registered [ 5283.246311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5292.601894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5292.664072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5294.252812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5300.103915] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5304.828653] LustreError: 175996:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5304.901294] LustreError: 175996:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5305.899829] LustreError: 175997:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5310.951455] LustreError: 175996:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5311.451714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5312.182329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5317.963966] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5327.059235] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5335.155796] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5343.939730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5344.299984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5351.540914] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5358.294929] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5365.805571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5375.969643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5375.979945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5375.986591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5376.015113] Lustre: Skipped 1 previous similar message [ 5381.094416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5381.112036] Lustre: Skipped 1 previous similar message [ 5381.917236] Lustre: server umount lustre-OST0000 complete [ 5386.239981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5386.242167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5386.266325] Lustre: Skipped 1 previous similar message [ 5386.278458] Lustre: Skipped 2 previous similar messages [ 5391.331239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5391.340926] Lustre: Skipped 1 previous similar message [ 5391.751080] Lustre: server umount lustre-MDT0000 complete [ 5395.094319] LustreError: 175977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847871 with bad export cookie 14586357746074650760 [ 5395.095189] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5395.104149] LustreError: 175977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5395.446318] Lustre: server umount lustre-MDT0001 complete [ 5405.855314] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5408.502100] Key type lgssc unregistered [ 5408.756790] LNet: 179557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5408.771668] LNetError: 179557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5408.799871] LNet: Removed LNI 192.168.203.152@tcp [ 5409.665283] Key type .llcrypt unregistered [ 5409.669501] Key type ._llcrypt unregistered [ 5431.949522] Key type ._llcrypt registered [ 5431.952237] Key type .llcrypt registered [ 5432.077660] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5433.642637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5433.669508] alg: No test for adler32 (adler32-zlib) [ 5434.655255] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 5434.908376] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5436.592897] Key type lgssc registered [ 5437.952215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5448.995094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5449.021172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5450.360653] Lustre: *** cfs_fail_loc=135, val=0*** [ 5450.446752] LustreError: 181082:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5450.460374] LustreError: 181082:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.152@tcp: cfg command failed: rc = -2 [ 5450.474274] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5450.474274] [ 5450.485579] LustreError: MGC192.168.203.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5450.502263] LustreError: 181043:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5450.522378] LustreError: 181043:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5450.532378] LustreError: 181043:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5450.554657] LustreError: 181043:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5450.688676] Lustre: server umount lustre-MDT0000 complete [ 5450.694397] LustreError: 181043:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5457.057708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5457.640406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5462.206976] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5468.135107] LustreError: 181454:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5468.156873] LustreError: 181454:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5470.706153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5471.043361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5475.669165] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5483.378654] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5488.741651] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5491.683859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5491.689575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5491.700494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5492.719461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5492.729805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5492.763195] Lustre: Skipped 1 previous similar message [ 5495.846082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5495.865128] Lustre: Skipped 1 previous similar message [ 5496.874966] Lustre: server umount lustre-MDT0000 complete [ 5500.873795] LustreError: 181435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847977 with bad export cookie 3317773681186578115 [ 5500.887317] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5500.895996] LustreError: 181435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5501.342337] Lustre: server umount lustre-MDT0001 complete [ 5512.318912] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5515.060023] Key type lgssc unregistered [ 5515.386652] LNet: 183777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5515.394480] LNetError: 183777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5515.428562] LNet: Removed LNI 192.168.203.152@tcp [ 5516.084862] Key type .llcrypt unregistered [ 5516.086952] Key type ._llcrypt unregistered [ 5537.522462] Key type ._llcrypt registered [ 5537.525373] Key type .llcrypt registered [ 5537.614593] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5537.934831] Key type .llcrypt unregistered [ 5537.937910] Key type ._llcrypt unregistered [ 5553.828968] Key type ._llcrypt registered [ 5553.832139] Key type .llcrypt registered [ 5553.918192] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5554.971770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5555.017462] alg: No test for adler32 (adler32-zlib) [ 5556.054204] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 5556.286269] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5557.960236] Key type lgssc registered [ 5558.975200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5567.039933] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5567.097810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5582.816696] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5587.936656] LustreError: 185912:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 5614.624213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5620.898537] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5627.654386] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5633.989899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5634.473604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5634.550860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5639.280949] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5639.653770] LustreError: 185937:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5639.671648] LustreError: 185937:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5644.772192] LustreError: 185938:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5646.603288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5646.865455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5650.899380] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5657.629114] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5662.828356] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5665.193760] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5680.625732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5680.626463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5680.655877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5680.660790] Lustre: Skipped 1 previous similar message [ 5683.875770] Lustre: server umount lustre-OST0000 complete [ 5687.777507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5687.787358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5687.801305] Lustre: Skipped 1 previous similar message [ 5687.808115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5690.851655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5690.859165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5693.956338] Lustre: server umount lustre-MDT0000 complete [ 5695.970872] LustreError: 186949:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5695.990853] LustreError: 186949:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5697.720175] LustreError: 188572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848174 with bad export cookie 5293148157057931029 [ 5697.732627] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5697.741531] LustreError: 188572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5698.137926] Lustre: server umount lustre-MDT0001 complete [ 5708.893931] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5711.769253] Key type lgssc unregistered [ 5712.049849] LNet: 189927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5712.059675] LNetError: 189927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5712.076643] LNet: Removed LNI 192.168.203.152@tcp [ 5712.890855] Key type .llcrypt unregistered [ 5712.893607] Key type ._llcrypt unregistered [ 5735.724053] Key type ._llcrypt registered [ 5735.727419] Key type .llcrypt registered [ 5735.863562] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5737.195774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5737.228293] alg: No test for adler32 (adler32-zlib) [ 5738.290388] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 5738.484949] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5740.136182] Key type lgssc registered [ 5741.170528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5747.817458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5747.836307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5749.218968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5752.671936] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5759.457939] LustreError: 191343:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.476725] LustreError: 191343:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5760.520597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5760.836783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5765.251823] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5772.701789] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5778.531325] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5785.412701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5785.640027] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5785.812347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5794.037464] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5794.295329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5803.868557] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5809.698471] Lustre: Failing over lustre-MDT0000 [ 5809.935473] Lustre: server umount lustre-MDT0000 complete [ 5811.193336] LustreError: 191337:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5812.195406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5812.206508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5812.229671] LustreError: 191343:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5814.753800] LustreError: 191342:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5814.756963] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5814.781810] LustreError: 191342:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5814.814733] Lustre: Skipped 1 previous similar message [ 5819.874301] LustreError: 192070:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5819.905986] LustreError: 192070:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 5830.086391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.119180] LustreError: 191339:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5830.130675] LustreError: 191339:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 5830.225683] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5830.443923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.510191] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5831.627351] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5834.852273] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5835.751258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5835.773946] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5835.820612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5839.737437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5848.957981] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5857.765318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5857.774553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5857.788418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5859.311619] Lustre: server umount lustre-OST0000 complete [ 5866.467789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5866.468256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5866.477828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5866.495775] Lustre: Skipped 1 previous similar message [ 5871.589061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5871.597769] Lustre: Skipped 2 previous similar messages [ 5872.765228] Lustre: server umount lustre-MDT0000 complete [ 5876.310068] LustreError: 191324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848353 with bad export cookie 3444363072692763303 [ 5876.311250] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5876.325340] LustreError: 191324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5876.760818] Lustre: server umount lustre-MDT0001 complete [ 5887.499803] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5890.405140] Key type lgssc unregistered [ 5890.727555] LNet: 196282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5890.735176] LNetError: 196282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5890.762826] LNet: Removed LNI 192.168.203.152@tcp [ 5891.621853] Key type .llcrypt unregistered [ 5891.624357] Key type ._llcrypt unregistered [ 5915.738450] Key type ._llcrypt registered [ 5915.744425] Key type .llcrypt registered [ 5915.867304] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5916.985612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5917.004956] alg: No test for adler32 (adler32-zlib) [ 5918.226302] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 5918.525687] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5920.200752] Key type lgssc registered [ 5921.341493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5929.484987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5929.506453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.844466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5935.902722] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5940.197035] LustreError: 197695:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5940.217628] LustreError: 197695:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5944.022547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.491830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5948.665924] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5956.200918] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5961.536987] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5967.602641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5968.088786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5974.015557] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5975.306928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5982.425651] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5986.464703] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5989.692031] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6006.441359] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6021.610664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6021.615373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6021.619319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6021.620100] LustreError: Skipped 1 previous similar message [ 6021.642640] Lustre: Skipped 1 previous similar message [ 6023.897622] Lustre: server umount lustre-OST0000 complete [ 6031.844192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6031.845078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6031.864459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6031.864470] Lustre: Skipped 1 previous similar message [ 6031.890129] Lustre: Skipped 1 previous similar message [ 6033.735221] Lustre: server umount lustre-MDT0000 complete [ 6036.965802] LustreError: 198423:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6036.991715] LustreError: 198423:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6037.410308] LustreError: 199543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848514 with bad export cookie 16931494686548673762 [ 6037.417283] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6037.428081] LustreError: 199543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6037.844828] Lustre: server umount lustre-MDT0001 complete [ 6047.555876] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6050.151789] Key type lgssc unregistered [ 6050.462523] LNet: 201570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6050.471654] LNetError: 201570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6050.485492] LNet: Removed LNI 192.168.203.152@tcp [ 6051.263830] Key type .llcrypt unregistered [ 6051.271799] Key type ._llcrypt unregistered [ 6067.031748] Key type ._llcrypt registered [ 6067.034327] Key type .llcrypt registered [ 6067.159597] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6068.285567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6068.359507] alg: No test for adler32 (adler32-zlib) [ 6069.371212] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 6069.614066] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6071.336179] Key type lgssc registered [ 6072.371100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6079.627342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6079.643860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6081.134636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6085.298971] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6090.729186] LustreError: 202819:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6090.745026] LustreError: 202819:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6092.950547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6093.337616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6097.656252] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6104.519453] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6109.967963] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6116.575798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6117.013722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6123.158695] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6125.249927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6130.817299] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6135.815816] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6145.891587] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6155.328195] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6169.084577] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6180.840990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6180.852165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.887606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6181.860109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6181.863605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6181.902460] Lustre: Skipped 1 previous similar message [ 6185.707692] Lustre: server umount lustre-OST0000 complete [ 6190.562121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6190.569708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6190.592990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6197.217830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6197.226277] Lustre: Skipped 3 previous similar messages [ 6202.338180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6202.350277] Lustre: Skipped 1 previous similar message [ 6204.384594] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6204.666589] Lustre: server umount lustre-MDT0000 complete [ 6206.445869] LustreError: 203547:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6206.475526] LustreError: 203547:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6208.307451] LustreError: 206636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848685 with bad export cookie 1110458173154429254 [ 6208.312708] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6208.320669] LustreError: 206636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6208.594193] Lustre: server umount lustre-MDT0001 complete [ 6218.323476] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6221.013096] Key type lgssc unregistered [ 6221.372827] LNet: 207889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6221.379046] LNetError: 207889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6221.399467] LNet: Removed LNI 192.168.203.152@tcp [ 6222.172855] Key type .llcrypt unregistered [ 6222.175393] Key type ._llcrypt unregistered [ 6244.309292] Key type ._llcrypt registered [ 6244.311468] Key type .llcrypt registered [ 6244.460504] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6245.676926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6245.724740] alg: No test for adler32 (adler32-zlib) [ 6246.776563] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 6247.096916] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6248.776582] Key type lgssc registered [ 6249.829408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6256.649823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6256.682622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6258.107292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6261.739464] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6267.363278] LustreError: 209301:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6267.393163] LustreError: 209301:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6269.752667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6270.062363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6274.792734] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6283.720824] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6289.502892] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6297.720114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6298.184845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6302.293409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6305.341485] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6313.232630] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6327.780841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6327.791480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6327.809779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6330.861294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6330.864699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6330.864856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6330.880419] Lustre: Skipped 1 previous similar message [ 6334.075349] Lustre: server umount lustre-OST0000 complete [ 6341.090326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6341.102930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6341.108292] Lustre: Skipped 1 previous similar message [ 6341.119618] Lustre: Skipped 1 previous similar message [ 6345.008520] Lustre: server umount lustre-MDT0000 complete [ 6346.213634] LustreError: 210029:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6346.250697] LustreError: 210029:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6349.299277] LustreError: 209282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848826 with bad export cookie 9979271305460386262 [ 6349.305225] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6349.310152] LustreError: 209282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6349.613335] Lustre: server umount lustre-MDT0001 complete [ 6364.250116] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6367.975948] Key type lgssc unregistered [ 6368.363902] LNet: 212989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6368.372938] LNetError: 212989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6369.448777] LNet: Removed LNI 192.168.203.152@tcp [ 6370.324264] Key type .llcrypt unregistered [ 6370.328230] Key type ._llcrypt unregistered [ 6390.287576] Key type ._llcrypt registered [ 6390.289604] Key type .llcrypt registered [ 6390.396274] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6391.235177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6391.409262] alg: No test for adler32 (adler32-zlib) [ 6392.462260] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 6392.662093] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6394.312817] Key type lgssc registered [ 6395.612391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6402.651457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6402.669465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6404.222186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6408.541139] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6414.819346] LustreError: 214255:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6414.842946] LustreError: 214255:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6416.250240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6416.661431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6420.567902] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6427.059926] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6433.125168] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6439.868945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6440.186902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6445.293382] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6450.447533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6453.133120] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6471.140788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6471.141387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6471.161159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6476.259490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6476.279718] Lustre: Skipped 2 previous similar messages [ 6477.399035] Lustre: server umount lustre-OST0000 complete [ 6483.429370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6483.443385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.466214] Lustre: Skipped 1 previous similar message [ 6483.497358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6486.501060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6486.501943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6489.001511] Lustre: server umount lustre-MDT0000 complete [ 6491.623609] LustreError: 214983:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6491.681461] LustreError: 214983:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6494.678215] LustreError: 217187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848971 with bad export cookie 100773859804443660 [ 6494.680248] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6494.697997] LustreError: 217187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6495.423515] Lustre: server umount lustre-MDT0001 complete [ 6507.834323] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6510.917829] Key type lgssc unregistered [ 6511.234659] LNet: 217942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6511.246916] LNetError: 217942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6511.273801] LNet: Removed LNI 192.168.203.152@tcp [ 6512.321197] Key type .llcrypt unregistered [ 6512.327255] Key type ._llcrypt unregistered [ 6532.566129] Key type ._llcrypt registered [ 6532.574801] Key type .llcrypt registered [ 6532.723103] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6533.963666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6534.017426] alg: No test for adler32 (adler32-zlib) [ 6535.416588] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 6535.758897] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6537.432743] Key type lgssc registered [ 6538.934417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6548.354569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6548.378117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6549.979242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6554.906177] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6559.203931] LustreError: 219210:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6559.243871] LustreError: 219210:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6564.325207] LustreError: 219211:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6564.872400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6565.122891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6569.930657] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6577.626464] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6584.082919] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6591.269258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6591.774721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6594.883420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6597.742303] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6606.563863] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6615.656629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6615.720689] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6616.061827] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6621.981751] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6629.713326] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6631.104627] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6642.737251] Lustre: Permanently deactivating lustre-OST0001 [ 6642.754766] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6651.465287] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6662.042219] Lustre: Permanently reactivating lustre-OST0001 [ 6662.076958] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6662.084870] Lustre: Skipped 2 previous similar messages [ 6668.873932] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6668.888183] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6668.896141] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6668.909146] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6668.965793] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6672.102298] Lustre: server umount lustre-OST0001 complete [ 6674.405645] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6674.408636] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6674.411158] LustreError: Skipped 1 previous similar message [ 6674.418064] LustreError: 221032:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6674.432133] Lustre: Skipped 2 previous similar messages [ 6674.455978] LustreError: 221032:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6675.940770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6675.948527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6675.964765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6676.970172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6676.979540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6677.629436] Lustre: server umount lustre-OST0000 complete [ 6683.109404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6683.117651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6683.128779] Lustre: Skipped 1 previous similar message [ 6683.138462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6685.666499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6685.674940] Lustre: Skipped 1 previous similar message [ 6686.863355] Lustre: server umount lustre-MDT0000 complete [ 6690.097170] LustreError: 219190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849166 with bad export cookie 5012734246690246173 [ 6690.099610] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6690.113823] LustreError: 219190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6690.542537] Lustre: server umount lustre-MDT0001 complete [ 6701.990477] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6704.943258] Key type lgssc unregistered [ 6705.324346] LNet: 224306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6705.344172] LNetError: 224306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6706.422959] LNet: Removed LNI 192.168.203.152@tcp [ 6707.795060] Key type .llcrypt unregistered [ 6707.798361] Key type ._llcrypt unregistered [ 6752.848682] Key type ._llcrypt registered [ 6752.851626] Key type .llcrypt registered [ 6752.969403] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6754.219957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6754.255353] alg: No test for adler32 (adler32-zlib) [ 6755.400437] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 6755.730356] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6757.456206] Key type lgssc registered [ 6758.565810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6765.821289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6765.844687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.169098] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6767.216498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6767.459543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6772.174531] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6780.613937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6780.769934] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6780.793846] Lustre: 226515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6781.139109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6785.679573] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6793.210374] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6798.930422] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6805.189980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6805.499742] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6805.811829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6812.413579] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6813.355204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6820.039509] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6823.441797] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6827.839054] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6835.524213] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6843.368854] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6856.812876] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6856.818117] Lustre: Skipped 1 previous similar message [ 6882.024556] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6882.032619] Lustre: Skipped 2 previous similar messages [ 6923.996878] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6924.012086] Lustre: Skipped 4 previous similar messages [ 6990.919217] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6990.924334] Lustre: Skipped 7 previous similar messages [ 6997.474680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6997.484758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6997.493911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6997.988982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6997.995815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6998.030568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7003.121358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7003.129235] Lustre: Skipped 1 previous similar message [ 7008.231114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7008.237541] Lustre: Skipped 1 previous similar message [ 7011.296189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7011.415728] Lustre: server umount lustre-OST0000 complete [ 7016.931784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7016.945689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7016.959566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7020.859479] Lustre: server umount lustre-MDT0000 complete [ 7024.661805] LustreError: 229190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849501 with bad export cookie 15221665508857785064 [ 7024.670128] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7024.674843] LustreError: 229190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7024.897387] Lustre: server umount lustre-MDT0001 complete [ 7040.000458] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 7044.313305] Key type lgssc unregistered [ 7044.664638] LNet: 231335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7044.689093] LNetError: 231335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7045.734665] LNet: Removed LNI 192.168.203.152@tcp [ 7046.781540] Key type .llcrypt unregistered [ 7046.786736] Key type ._llcrypt unregistered [ 7080.842810] Key type ._llcrypt registered [ 7080.844595] Key type .llcrypt registered [ 7080.950972] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 7082.853672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7082.904156] alg: No test for adler32 (adler32-zlib) [ 7084.218325] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 7084.587150] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 7086.360535] Key type lgssc registered [ 7088.273525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7097.080470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7097.118089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7098.860491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7103.144191] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7109.091677] LustreError: 232755:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7109.117887] LustreError: 232755:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7111.422608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7111.771127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7116.117856] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7122.687464] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7128.743620] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7135.911696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7136.231341] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 7136.333165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7141.776904] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7143.442498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7149.200399] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7166.946862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7166.953816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7166.963772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7168.975092] Lustre: server umount lustre-OST0000 complete [ 7173.090601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7173.102942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7173.116732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7174.122613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7179.235827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7179.249620] Lustre: Skipped 3 previous similar messages [ 7182.427495] Lustre: server umount lustre-MDT0000 complete [ 7183.336340] LustreError: 233768:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7183.347160] LustreError: 233768:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7185.979084] LustreError: 232735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849662 with bad export cookie 1967515833707460608 [ 7185.985973] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7185.996594] LustreError: 232735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7186.378334] Lustre: server umount lustre-MDT0001 complete [ 7197.059718] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 7199.606784] Key type lgssc unregistered [ 7199.956213] LNet: 236448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7199.970330] LNetError: 236448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7200.016578] LNet: Removed LNI 192.168.203.152@tcp [ 7201.056810] Key type .llcrypt unregistered [ 7201.058707] Key type ._llcrypt unregistered [ 7222.684655] Key type ._llcrypt registered [ 7222.687654] Key type .llcrypt registered [ 7222.784130] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 7223.696171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7223.851073] alg: No test for adler32 (adler32-zlib) [ 7224.963460] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 7225.184091] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 7226.864253] Key type lgssc registered [ 7228.139372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7497.263990] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 7500.341792] Key type lgssc unregistered [ 7500.612656] LNet: 238474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7500.623129] LNetError: 238474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7500.648482] LNet: Removed LNI 192.168.203.152@tcp [ 7501.615440] Key type .llcrypt unregistered [ 7501.617105] Key type ._llcrypt unregistered [ 7530.864420] Key type ._llcrypt registered [ 7530.866824] Key type .llcrypt registered [ 7531.040952] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 7547.372615] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 7548.479869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7548.577675] alg: No test for adler32 (adler32-zlib) [ 7549.785628] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 7550.053560] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 7551.712395] Key type lgssc registered [ 7553.249326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7585.497167] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7316 sec ========== 13:00:59 (1787850059) [ 7587.476831] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:01:01 (1787850061) === [ 7591.061592] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:01:05 (1787850065) === [ 7635.706781] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 7638.465556] Key type lgssc unregistered [ 7638.747633] LNet: 243729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7638.762107] LNetError: 243729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7638.790498] LNet: Removed LNI 192.168.203.152@tcp [ 7639.682522] Key type .llcrypt unregistered [ 7639.685346] Key type ._llcrypt unregistered