[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1041108614 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003753] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005026] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.011149] LSM: Security Framework initializing [ 0.012075] Yama: becoming mindful. [ 0.013051] SELinux: Initializing. [ 0.014398] *** VALIDATE selinux *** [ 0.030347] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.039103] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.041080] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.042396] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.043281] *** VALIDATE tmpfs *** [ 0.045470] *** VALIDATE proc *** [ 0.047040] *** VALIDATE cgroup *** [ 0.048014] *** VALIDATE cgroup2 *** [ 0.049354] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.050615] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.051012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.052036] Spectre V2 : User space: Vulnerable [ 0.053019] Speculative Store Bypass: Vulnerable [ 0.060259] debug: unmapping init [mem 0xffffffff85259000-0xffffffff85260fff] [ 0.063462] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.066065] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.067028] ... version: 2 [ 0.068018] ... bit width: 48 [ 0.069015] ... generic registers: 4 [ 0.070015] ... value mask: 0000ffffffffffff [ 0.071017] ... max period: 00007fffffffffff [ 0.072161] ... fixed-purpose events: 3 [ 0.073014] ... event mask: 000000070000000f [ 0.074692] rcu: Hierarchical SRCU implementation. [ 0.078898] smp: Bringing up secondary CPUs ... [ 0.080189] x86: Booting SMP configuration: [ 0.081161] .... node #0, CPUs: #1 #2 #3 [ 0.106177] smp: Brought up 1 node, 4 CPUs [ 0.108025] smpboot: Max logical packages: 1 [ 0.109019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.220770] node 0 deferred pages initialised in 108ms [ 0.233123] devtmpfs: initialized [ 0.235708] x86/mm: Memory block size: 128MB [ 0.239468] gcov: version magic: 0x41383552 [ 0.241854] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.242974] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243681] pinctrl core: initialized pinctrl subsystem [ 0.244223] [ 0.245010] ************************************************************* [ 0.246015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247013] ** ** [ 0.248014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.249020] ** ** [ 0.250018] ** This means that this kernel is built to expose internal ** [ 0.251018] ** IOMMU data structures, which may compromise security on ** [ 0.252019] ** your system. ** [ 0.253022] ** ** [ 0.254020] ** If you see this message and you are not debugging the ** [ 0.255020] ** kernel, report this immediately to your vendor! ** [ 0.256265] ** ** [ 0.257023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258031] ************************************************************* [ 0.259975] NET: Registered protocol family 16 [ 0.260757] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.261100] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.262132] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.264344] cpuidle: using governor menu [ 0.267171] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.268933] PCI: Using configuration type 1 for base access [ 0.269144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.284000] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.288154] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.301162] cryptd: max_cpu_qlen set to 1000 [ 0.346512] ACPI: Added _OSI(Module Device) [ 0.347000] ACPI: Added _OSI(Processor Device) [ 0.349029] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.351032] ACPI: Added _OSI(Processor Aggregator Device) [ 0.358891] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.366958] ACPI: Interpreter enabled [ 0.369078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.371012] ACPI: Using IOAPIC for interrupt routing [ 0.373152] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.377572] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.392867] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.394063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.398026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.402098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.408111] acpiphp: Slot [2] registered [ 0.410239] acpiphp: Slot [5] registered [ 0.412202] acpiphp: Slot [6] registered [ 0.414231] acpiphp: Slot [7] registered [ 0.416148] acpiphp: Slot [8] registered [ 0.418155] acpiphp: Slot [9] registered [ 0.420155] acpiphp: Slot [10] registered [ 0.422182] acpiphp: Slot [3] registered [ 0.424146] acpiphp: Slot [4] registered [ 0.426158] acpiphp: Slot [11] registered [ 0.428117] acpiphp: Slot [12] registered [ 0.430109] acpiphp: Slot [13] registered [ 0.431182] acpiphp: Slot [14] registered [ 0.433126] acpiphp: Slot [15] registered [ 0.435120] acpiphp: Slot [16] registered [ 0.437142] acpiphp: Slot [17] registered [ 0.439146] acpiphp: Slot [18] registered [ 0.441133] acpiphp: Slot [19] registered [ 0.442183] acpiphp: Slot [20] registered [ 0.444114] acpiphp: Slot [21] registered [ 0.446130] acpiphp: Slot [22] registered [ 0.448229] acpiphp: Slot [23] registered [ 0.450133] acpiphp: Slot [24] registered [ 0.451179] acpiphp: Slot [25] registered [ 0.453110] acpiphp: Slot [26] registered [ 0.455120] acpiphp: Slot [27] registered [ 0.457111] acpiphp: Slot [28] registered [ 0.459220] acpiphp: Slot [29] registered [ 0.460115] acpiphp: Slot [30] registered [ 0.462114] acpiphp: Slot [31] registered [ 0.464097] PCI host bridge to bus 0000:00 [ 0.466030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.469023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.472021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.474026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.477035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.481032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.483315] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.488769] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.492589] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.504993] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.510061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.513024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.515031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.519018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.521690] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.524999] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.528047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.532155] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.538012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.554032] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.561014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.569169] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.579017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.587016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.612016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.633852] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.642015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.652021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.678017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.693310] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.704016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.713028] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.769022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.781812] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.791017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.801018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.825019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.841727] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.851016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.862022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.881017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.892661] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.901018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.911033] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.958021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.975288] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.977407] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.980465] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.983430] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.986270] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.991116] iommu: Default domain type: Passthrough [ 0.994051] SCSI subsystem initialized [ 0.996269] ACPI: bus type USB registered [ 0.998274] usbcore: registered new interface driver usbfs [ 1.001088] usbcore: registered new interface driver hub [ 1.004124] usbcore: registered new device driver usb [ 1.006273] pps_core: LinuxPPS API ver. 1 registered [ 1.008012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.012065] PTP clock support registered [ 1.015315] EDAC MC: Ver: 3.0.0 [ 1.017887] PCI: Using ACPI for IRQ routing [ 1.019096] NetLabel: Initializing [ 1.021030] NetLabel: domain hash size = 128 [ 1.024032] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.026119] NetLabel: unlabeled traffic allowed by default [ 1.030092] vgaarb: loaded [ 1.034563] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.037013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.044256] clocksource: Switched to clocksource kvm-clock [ 1.208310] VFS: Disk quotas dquot_6.6.0 [ 1.210220] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.213221] *** VALIDATE ramfs *** [ 1.214637] *** VALIDATE hugetlbfs *** [ 1.216674] pnp: PnP ACPI init [ 1.220611] pnp: PnP ACPI: found 6 devices [ 1.241324] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.245766] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.250029] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.252667] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.256583] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.259683] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.264350] NET: Registered protocol family 2 [ 1.268356] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.278058] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.285252] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.295064] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.299702] TCP: Hash tables configured (established 65536 bind 65536) [ 1.303630] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.307746] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.311946] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.316366] NET: Registered protocol family 1 [ 1.319661] RPC: Registered named UNIX socket transport module. [ 1.322305] RPC: Registered udp transport module. [ 1.324636] RPC: Registered tcp transport module. [ 1.327087] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.330219] NET: Registered protocol family 44 [ 1.333416] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.337898] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.340316] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.344410] PCI: CLS 0 bytes, default 64 [ 1.347678] Unpacking initramfs... [ 4.228590] debug: unmapping init [mem 0xffff97a0fcc54000-0xffff97a0fffbffff] [ 4.232458] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.234514] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.237840] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.687077] Initialise system trusted keyrings [ 5.688959] Key type blacklist registered [ 5.692161] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.714818] zbud: loaded [ 5.718305] *** VALIDATE nfs *** [ 5.719937] *** VALIDATE nfs4 *** [ 5.721853] pstore: using deflate compression [ 5.726125] Platform Keyring initialized [ 5.867332] NET: Registered protocol family 38 [ 5.868708] Key type asymmetric registered [ 5.870095] Asymmetric key parser 'x509' registered [ 5.871871] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.875252] io scheduler mq-deadline registered [ 5.876838] io scheduler kyber registered [ 5.878491] io scheduler bfq registered [ 5.880303] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.885425] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.887917] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.890683] ACPI: Power Button [PWRF] [ 5.896499] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.903286] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.922982] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.933689] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.954555] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.984352] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.016557] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.021460] Non-volatile memory driver v1.3 [ 6.023108] Linux agpgart interface v0.103 [ 6.053490] virtio_blk virtio1: [vda] 134696 512-byte logical blocks (69.0 MB/65.8 MiB) [ 6.056544] vda: detected capacity change from 0 to 68964352 [ 6.073767] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.077827] vdb: detected capacity change from 0 to 1073741824 [ 6.094631] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.097919] vdc: detected capacity change from 0 to 2621440000 [ 6.114707] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.117946] vdd: detected capacity change from 0 to 2621440000 [ 6.139624] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.142693] vde: detected capacity change from 0 to 4294967296 [ 6.158460] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.161735] vdf: detected capacity change from 0 to 4294967296 [ 6.170794] libphy: Fixed MDIO Bus: probed [ 6.186320] usbcore: registered new interface driver usbserial_generic [ 6.190876] usbserial: USB Serial support registered for generic [ 6.194069] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.200305] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.201982] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.204957] mousedev: PS/2 mouse device common for all mice [ 6.208939] rtc_cmos 00:05: RTC can wake from S4 [ 6.212259] rtc_cmos 00:05: registered as rtc0 [ 6.214577] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.217046] intel_pstate: CPU model not supported [ 6.221253] hid: raw HID events driver (C) Jiri Kosina [ 6.224065] usbcore: registered new interface driver usbhid [ 6.226112] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.229474] usbhid: USB HID core driver [ 6.239590] drop_monitor: Initializing network drop monitor service [ 6.242432] Initializing XFRM netlink socket [ 6.244608] NET: Registered protocol family 10 [ 6.248246] Segment Routing with IPv6 [ 6.249983] NET: Registered protocol family 17 [ 6.251629] mpls_gso: MPLS GSO support [ 6.256119] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.266648] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.297839] RAS: Correctable Errors collector initialized. [ 6.348784] AVX version of gcm_enc/dec engaged. [ 6.350983] AES CTR mode by8 optimization enabled [ 6.503892] sched_clock: Marking stable (6503798256, 0)->(8167187252, -1663388996) [ 6.509806] registered taskstats version 1 [ 6.512879] Loading compiled-in X.509 certificates [ 6.516068] zswap: loaded using pool lzo/zbud [ 6.555890] Key type big_key registered [ 6.575619] Key type encrypted registered [ 6.577289] ima: No TPM chip found, activating TPM-bypass! [ 6.580543] ima: Allocated hash algorithm: sha1 [ 6.583198] ima: No architecture policies found [ 6.585731] evm: Initialising EVM extended attributes: [ 6.588269] evm: security.selinux [ 6.590200] evm: security.ima [ 6.591967] evm: security.capability [ 6.593971] evm: HMAC attrs: 0x1 [ 6.597664] rtc_cmos 00:05: setting system clock to 2026-03-02 16:35:29 UTC (1772469329) [ 6.609795] debug: unmapping init [mem 0xffffffff86203000-0xffffffff863fffff] [ 6.616503] debug: unmapping init [mem 0xffffffff84f82000-0xffffffff85258fff] [ 6.625174] Write protecting the kernel read-only data: 28672k [ 6.632287] debug: unmapping init [mem 0xffffffff83603000-0xffffffff837fffff] [ 6.636782] debug: unmapping init [mem 0xffffffff83f14000-0xffffffff83ffffff] [ 6.773507] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.785239] systemd[1]: Detected virtualization kvm. [ 6.788481] systemd[1]: Detected architecture x86-64. [ 6.794248] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.018776] systemd[1]: No hostname configured. [ 7.021711] systemd[1]: Set hostname to . [ 7.025110] random: systemd: uninitialized urandom read (16 bytes read) [ 7.032310] systemd[1]: Initializing machine ID from random generator. [ 7.280355] random: ln: uninitialized urandom read (6 bytes read) [ 7.451793] random: systemd: uninitialized urandom read (16 bytes read) [ 7.455865] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 7.474336] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 7.490956] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.070834] device-mapper: uevent: version 1.0.3 [ 11.073445] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 13.437536] random: fast init done [ 13.482572] virtio_net virtio0 ens2: renamed from eth0 [ 14.274754] scsi host0: ata_piix [ 14.397473] scsi host1: ata_piix [ 14.401100] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 14.405472] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 19.223272] random: crng init done [ 19.224945] random: 7 urandom warning(s) missed due to ratelimiting [ 22.637415] dracut-initqueue[598]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ 23.881735] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 28.504254] printk: systemd: 26 output lines suppressed due to ratelimiting [ 29.285511] SELinux: Disabled at runtime. [ 29.396937] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 29.415863] systemd[1]: Detected virtualization kvm. [ 29.418723] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 31.039561] systemd[1]: initrd-switch-root.service: Succeeded. [ 31.044989] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 31.057860] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 31.065493] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 31.070535] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 31.086913] systemd[1]: Starting Journal Service... Starting Journal Service... [ 31.125891] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK [[ 31.221106] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 33.816298] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 35.365285] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 36.253662] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 37.943173] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 38.054223] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 41.940559] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 42.812917] NFS: Registering the id_resolver key type [ 42.815486] Key type id_resolver registered [ 42.817509] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg240-server login: [ 91.183831] libcfs: loading out-of-tree module taints kernel. [ 91.218257] Key type ._llcrypt registered [ 91.220620] Key type .llcrypt registered [ 91.308829] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 104.143824] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 105.353555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.362983] alg: No test for adler32 (adler32-zlib) [ 106.545745] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 107.085892] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 108.751588] Key type lgssc registered [ 109.886160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 120.008553] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 150.569947] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 158.856736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 158.879842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.100801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 160.121904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 160.214421] Lustre: lustre-MDT0000: new disk, initializing [ 160.274300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.292406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 162.450595] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.270479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.324460] Lustre: 6508:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 171.348893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 171.352345] Lustre: Skipped 1 previous similar message [ 171.402585] Lustre: lustre-MDT0001: new disk, initializing [ 171.445948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 171.477698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 171.496948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 175.171459] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.114827] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 187.003601] hrtimer: interrupt took 3263442 ns [ 201.711200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.982466] Lustre: lustre-OST0000: new disk, initializing [ 201.987248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.128356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.828345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.842871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 205.032652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 211.356409] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.395591] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.578365] Lustre: lustre-OST0001: new disk, initializing [ 234.587524] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 234.690092] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.751501] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.272030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 244.287195] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 244.341463] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 255.691450] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 262.839418] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 274.633674] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing check_logdir /tmp/testlogs/ [ 281.518752] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing yml_node [ 286.042606] Lustre: DEBUG MARKER: Client: 2.17.50.195 [ 288.540976] Lustre: DEBUG MARKER: MDS: 2.17.50.195 [ 291.645553] Lustre: DEBUG MARKER: OSS: 2.17.50.195 [ 293.936890] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Mar 2 11:40:14 EST 2026 [ 314.994708] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 317.364774] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 335.331543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 335.339626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.353149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 336.355459] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.358750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 336.369727] Lustre: Skipped 2 previous similar messages [ 336.399760] Lustre: Skipped 3 previous similar messages [ 341.739752] Lustre: server umount lustre-MDT0000 complete [ 345.775370] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772469668 with bad export cookie 7178035074527392795 [ 345.788178] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 346.598938] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.602674] LustreError: 8422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.614471] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.651807] LustreError: 8422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 350.623848] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 350.623862] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 350.632031] Lustre: Skipped 1 previous similar message [ 350.668467] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 352.383715] Lustre: server umount lustre-MDT0001 complete [ 357.125928] Lustre: server umount lustre-OST0000 complete [ 361.352389] Lustre: server umount lustre-OST0001 complete [ 369.284536] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 380.611890] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 423.258361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.726872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 423.806857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 423.966583] Lustre: lustre-MDT0000: new disk, initializing [ 424.101560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.144051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 430.851852] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 444.734155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.004053] Lustre: 14991:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 445.160788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 445.176260] Lustre: Skipped 1 previous similar message [ 445.236684] Lustre: lustre-MDT0001: new disk, initializing [ 445.317030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 445.365983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 445.385592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 451.498550] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 460.453533] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 465.262489] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 473.390384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 473.755281] Lustre: lustre-OST0000: new disk, initializing [ 473.765439] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 473.901377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 475.905070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 475.932220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 476.150422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 480.922632] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 490.402939] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 496.094079] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 496.453527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 502.159976] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 502.525818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 506.863171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 506.864860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.881840] LustreError: Skipped 1 previous similar message [ 506.884290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.897817] Lustre: Skipped 2 previous similar messages [ 506.927582] Lustre: Skipped 2 previous similar messages [ 511.635753] Lustre: server umount lustre-OST0000 complete [ 517.090294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 517.099320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.111869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 517.117918] Lustre: Skipped 1 previous similar message [ 517.141858] Lustre: Skipped 1 previous similar message [ 522.210484] LustreError: 14998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.233595] LustreError: 14998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 522.574662] Lustre: server umount lustre-MDT0000 complete [ 526.316098] LustreError: 15002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.369407] LustreError: 15002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 526.463772] LustreError: 14985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772469849 with bad export cookie 7178035074527394321 [ 526.464498] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 526.473617] LustreError: 14985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 527.109477] Lustre: server umount lustre-MDT0001 complete [ 532.677961] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 11:44:12 (1772469852) [ 540.764749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.491512] LustreError: 18853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.608716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 546.752745] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 551.915984] LustreError: 18854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.940279] LustreError: 18854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 557.681421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.370539] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 571.201649] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 576.177501] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 584.296852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 591.478374] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 598.024316] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 609.765270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 609.775907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.797169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 615.687328] Lustre: server umount lustre-OST0000 complete [ 626.910775] Lustre: server umount lustre-MDT0000 complete [ 627.681365] LustreError: 18853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 627.700398] LustreError: 18853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 631.334349] LustreError: 18834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772469954 with bad export cookie 7178035074527395203 [ 631.339187] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 631.351850] LustreError: 18834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 632.253845] Lustre: server umount lustre-MDT0001 complete [ 649.983904] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 654.426370] Key type lgssc unregistered [ 654.973619] LNet: 22055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.983556] LNetError: 22055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.013822] LNet: Removed LNI 192.168.202.140@tcp [ 656.727180] Key type .llcrypt unregistered [ 656.729052] Key type ._llcrypt unregistered [ 685.776076] Key type ._llcrypt registered [ 685.780453] Key type .llcrypt registered [ 685.950872] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 687.532491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 687.777059] alg: No test for adler32 (adler32-zlib) [ 688.967892] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 689.278168] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 691.138440] Key type lgssc registered [ 692.860755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 703.070272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 703.088933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.413382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 709.935283] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 714.725882] LustreError: 23460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 714.743941] LustreError: 23460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 719.842573] LustreError: 23459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 721.125770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.946020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 727.173698] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 734.535664] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 739.442811] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 748.037754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 748.475605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 749.518599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 756.632299] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 763.129910] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 784.681734] Lustre: server umount lustre-OST0000 complete [ 792.547249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.567124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 792.567532] Lustre: Skipped 1 previous similar message [ 792.579718] Lustre: Skipped 1 previous similar message [ 797.672722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 797.681743] Lustre: Skipped 1 previous similar message [ 799.892553] Lustre: server umount lustre-MDT0000 complete [ 802.789888] LustreError: 23460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 802.820351] LustreError: 23460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 803.807121] Lustre: 22770:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772470110/real 1772470110] req@00000000a7fa9ab0 x1858569515336832/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1772470126 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 803.810592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 803.829473] Lustre: 22770:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 804.380553] LustreError: 23439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772470127 with bad export cookie 16382455146478165039 [ 804.402406] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 804.417697] LustreError: 23439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 804.959436] Lustre: server umount lustre-MDT0001 complete [ 820.134914] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 825.044571] Key type lgssc unregistered [ 825.468946] LNet: 27086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.471767] LNetError: 27086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.482856] LNet: Removed LNI 192.168.202.140@tcp [ 826.203562] Key type .llcrypt unregistered [ 826.205150] Key type ._llcrypt unregistered [ 857.506506] Key type ._llcrypt registered [ 857.511560] Key type .llcrypt registered [ 857.684459] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 859.617225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 859.662558] alg: No test for adler32 (adler32-zlib) [ 860.907645] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 861.296082] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 863.135268] Key type lgssc registered [ 864.977595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 874.670852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 874.727792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 876.533690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 882.303485] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 886.755216] LustreError: 28485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 886.769559] LustreError: 28485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 891.877222] LustreError: 28486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 892.224404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.674745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 897.368595] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 904.758977] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 909.418650] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 925.754862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 926.057342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 933.282731] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 941.270775] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 947.653227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 952.807524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.844553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 952.847396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 952.867806] Lustre: Skipped 1 previous similar message [ 954.278743] Lustre: server umount lustre-OST0000 complete [ 959.463415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 959.480845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 959.507579] Lustre: Skipped 1 previous similar message [ 959.517093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 963.041968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 963.045459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 963.060948] Lustre: Skipped 1 previous similar message [ 968.170175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 968.173164] Lustre: Skipped 1 previous similar message [ 972.260218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 972.265260] Lustre: Skipped 1 previous similar message [ 973.282156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 973.573796] Lustre: server umount lustre-MDT0000 complete [ 977.390683] LustreError: 28485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 977.414773] LustreError: 28485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 978.052595] LustreError: 28465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772470300 with bad export cookie 12163543600444108413 [ 978.054771] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 978.073424] LustreError: 28465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 978.728704] Lustre: server umount lustre-MDT0001 complete [ 990.850705] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 994.272558] Key type lgssc unregistered [ 994.612984] LNet: 32017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.639804] LNetError: 32017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.691178] LNet: Removed LNI 192.168.202.140@tcp [ 996.564828] Key type .llcrypt unregistered [ 996.566874] Key type ._llcrypt unregistered [ 1022.668348] Key type ._llcrypt registered [ 1022.670552] Key type .llcrypt registered [ 1022.789572] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1024.022652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1024.175079] alg: No test for adler32 (adler32-zlib) [ 1025.418972] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 1025.855347] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 1027.575286] Key type lgssc registered [ 1029.051754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1038.674732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1038.705548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1040.371225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.398513] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1050.602433] LustreError: 33415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1050.627874] LustreError: 33415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1055.713802] LustreError: 33414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1056.692734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1057.219400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1062.139520] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1067.582727] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1071.809156] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1080.883385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1081.325544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1085.442154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1088.798597] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1096.358502] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1105.901659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.919033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1105.919958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1105.946701] Lustre: Skipped 1 previous similar message [ 1111.018179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1111.029618] Lustre: Skipped 1 previous similar message [ 1111.640357] Lustre: server umount lustre-OST0000 complete [ 1116.136554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.139616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1116.145592] Lustre: Skipped 2 previous similar messages [ 1116.165991] Lustre: Skipped 1 previous similar message [ 1121.252460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1121.264026] Lustre: Skipped 1 previous similar message [ 1122.704540] Lustre: server umount lustre-MDT0000 complete [ 1126.119129] LustreError: 33395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772470449 with bad export cookie 12119889263826822835 [ 1126.121977] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1126.135389] LustreError: 33395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1126.717656] Lustre: server umount lustre-MDT0001 complete [ 1139.195320] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 1142.593542] Key type lgssc unregistered [ 1143.045985] LNet: 36596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.065281] LNetError: 36596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.081916] LNet: Removed LNI 192.168.202.140@tcp [ 1143.906550] Key type .llcrypt unregistered [ 1143.909548] Key type ._llcrypt unregistered [ 1169.744194] Key type ._llcrypt registered [ 1169.746108] Key type .llcrypt registered [ 1169.892746] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1171.119846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1171.152404] alg: No test for adler32 (adler32-zlib) [ 1172.373967] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 1172.681056] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 1174.479375] Key type lgssc registered [ 1176.016871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1184.500833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1184.523788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1186.158947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1191.560275] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1196.515922] LustreError: 37991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.551372] LustreError: 37991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1200.679669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1201.011219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1205.530620] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1211.008270] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1215.398502] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1222.243426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1222.556609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.485269] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1231.790425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1234.328809] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1241.056869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1241.063269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.076346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1242.092441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1242.095157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1242.108293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.145204] Lustre: Skipped 2 previous similar messages [ 1246.222276] Lustre: server umount lustre-OST0000 complete [ 1267.683647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.689072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1267.712146] Lustre: Skipped 1 previous similar message [ 1273.779155] Lustre: server umount lustre-MDT0000 complete [ 1273.824304] LustreError: 38703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.050905] LustreError: 37971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772470600 with bad export cookie 14109938571548369815 [ 1278.065408] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1278.066118] LustreError: 37971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1284.694233] Lustre: server umount lustre-MDT0001 complete [ 1298.988207] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 1303.271818] Key type lgssc unregistered [ 1303.643866] LNet: 41348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1303.657409] LNetError: 41348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.676879] LNet: Removed LNI 192.168.202.140@tcp [ 1304.738865] Key type .llcrypt unregistered [ 1304.740692] Key type ._llcrypt unregistered [ 1331.533808] Key type ._llcrypt registered [ 1331.535976] Key type .llcrypt registered [ 1331.645946] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1332.586762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1332.610843] alg: No test for adler32 (adler32-zlib) [ 1334.037573] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 1334.301079] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 1336.007205] Key type lgssc registered [ 1337.629451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1345.887565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1345.911723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1347.486479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.139185] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1357.799480] LustreError: 42748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.820773] LustreError: 42748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1361.783293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.283655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1367.396503] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1372.980628] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1376.975618] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1384.191638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1384.501631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.070033] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1392.700386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1397.301736] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1405.158477] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.40@tcp (stopping) [ 1407.980032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.986761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1408.011235] Lustre: Skipped 2 previous similar messages [ 1408.034583] Lustre: Skipped 2 previous similar messages [ 1410.259650] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.40@tcp (stopping) [ 1413.091959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1413.108549] Lustre: Skipped 2 previous similar messages [ 1418.211222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1418.223702] Lustre: Skipped 3 previous similar messages [ 1419.231114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1419.546445] Lustre: server umount lustre-MDT0000 complete [ 1420.497538] LustreError: 42743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1423.346174] LustreError: 42745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1423.370827] LustreError: 42745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1424.301721] LustreError: 44364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772470747 with bad export cookie 11452371165870375026 [ 1424.316470] LustreError: 44364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1424.320415] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1425.608549] LustreError: 42743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1428.448768] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.476214] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1428.481133] Lustre: Skipped 1 previous similar message [ 1430.734405] LustreError: 44768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1430.758869] LustreError: 44768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1430.948546] Lustre: server umount lustre-MDT0001 complete [ 1447.351455] Lustre: server umount lustre-OST0000 complete [ 1467.890583] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 1471.557346] Key type lgssc unregistered [ 1471.863126] LNet: 46261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.887344] LNetError: 46261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1471.901626] LNet: Removed LNI 192.168.202.140@tcp [ 1472.742740] Key type .llcrypt unregistered [ 1472.747474] Key type ._llcrypt unregistered [ 1498.199528] Key type ._llcrypt registered [ 1498.208036] Key type .llcrypt registered [ 1498.410827] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1499.619501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1499.710562] alg: No test for adler32 (adler32-zlib) [ 1500.711697] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 1500.900845] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 1502.615526] Key type lgssc registered [ 1503.663662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1514.564515] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1514.591286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1530.144168] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1535.263480] LustreError: 47627:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.140@tcp: failed processing log, type 4: rc = -110 [ 1562.015644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.832852] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1576.779885] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1743.296723] Lustre: server umount lustre-OST0000 complete [ 1763.161198] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 1766.202601] Key type lgssc unregistered [ 1766.579985] LNet: 49186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.618899] LNetError: 49186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1766.637066] LNet: Removed LNI 192.168.202.140@tcp [ 1767.811299] Key type .llcrypt unregistered [ 1767.816805] Key type ._llcrypt unregistered [ 1795.328813] Key type ._llcrypt registered [ 1795.330868] Key type .llcrypt registered [ 1795.394561] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1796.163488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1796.231148] alg: No test for adler32 (adler32-zlib) [ 1797.434378] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 1797.770745] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 1799.503175] Key type lgssc registered [ 1801.274216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1811.207844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1811.220755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1812.736600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.768990] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1822.179091] LustreError: 50586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1822.200108] LustreError: 50586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1827.309679] LustreError: 50587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1827.773696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.505389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1834.325729] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1841.256519] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1845.757844] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1854.519152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1854.998465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.115097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1863.502990] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1871.296710] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1878.504725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1878.510442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1878.531418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1878.539464] Lustre: Skipped 1 previous similar message [ 1878.568136] Lustre: Skipped 1 previous similar message [ 1881.238081] Lustre: server umount lustre-OST0000 complete [ 1888.748976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1888.751379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1888.764833] Lustre: Skipped 1 previous similar message [ 1892.349219] Lustre: server umount lustre-MDT0000 complete [ 1893.856370] LustreError: 50587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1893.882113] LustreError: 50587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1898.158781] LustreError: 51306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772471221 with bad export cookie 10396490203392500093 [ 1898.162959] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1898.171508] LustreError: 51306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1899.090868] Lustre: server umount lustre-MDT0001 complete [ 1914.532396] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 1917.570961] Key type lgssc unregistered [ 1917.763799] LNet: 53766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.769733] LNetError: 53766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1917.787096] LNet: Removed LNI 192.168.202.140@tcp [ 1919.056059] Key type .llcrypt unregistered [ 1919.058322] Key type ._llcrypt unregistered [ 1945.786701] Key type ._llcrypt registered [ 1945.794845] Key type .llcrypt registered [ 1945.927923] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1947.542221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1947.587668] alg: No test for adler32 (adler32-zlib) [ 1948.694941] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 1948.939692] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 1950.687243] Key type lgssc registered [ 1951.928549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1960.511650] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1960.526532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1976.159777] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1981.279244] LustreError: 55135:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.140@tcp: failed processing log, type 4: rc = -110 [ 2008.031223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2015.480869] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2022.774741] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2031.694756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.585280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.695598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2037.741962] LustreError: 55161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2037.769518] LustreError: 55161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2038.633915] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2042.849239] LustreError: 55160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.876493] LustreError: 55160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2048.008414] LustreError: 56058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2048.053386] LustreError: 56058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2049.856932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.448866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2056.181089] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2063.159609] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2068.004410] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2071.013814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.014989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2071.028096] Lustre: Skipped 1 previous similar message [ 2071.030355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2071.072550] Lustre: Skipped 1 previous similar message [ 2076.130942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2076.154515] Lustre: Skipped 1 previous similar message [ 2081.256492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2081.269525] Lustre: Skipped 1 previous similar message [ 2085.343916] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2085.525666] Lustre: server umount lustre-OST0000 complete [ 2111.910509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2111.926445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2111.932981] Lustre: Skipped 1 previous similar message [ 2117.088840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2117.091589] Lustre: Skipped 2 previous similar messages [ 2125.279395] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2125.755984] Lustre: server umount lustre-MDT0000 complete [ 2127.349756] LustreError: 57535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.520230] LustreError: 55143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772471453 with bad export cookie 7632540180490525856 [ 2130.534371] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2137.026918] Lustre: server umount lustre-MDT0001 complete [ 2151.064894] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 2154.629547] Key type lgssc unregistered [ 2154.984307] LNet: 58535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2154.995496] LNetError: 58535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.015780] LNet: Removed LNI 192.168.202.140@tcp [ 2156.099529] Key type .llcrypt unregistered [ 2156.102071] Key type ._llcrypt unregistered [ 2180.770198] Key type ._llcrypt registered [ 2180.773736] Key type .llcrypt registered [ 2180.891985] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 2181.980030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2182.025520] alg: No test for adler32 (adler32-zlib) [ 2183.242904] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 2183.577424] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 2185.263389] Key type lgssc registered [ 2186.981498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2198.707754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2198.780811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.691271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.551987] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2210.799609] LustreError: 59931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.837796] LustreError: 59931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2215.913652] LustreError: 59930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.046443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.791944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2223.990540] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2231.587368] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2235.509426] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2243.173386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2243.502272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.556453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2250.944498] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2257.913544] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2278.368794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2278.371492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.404795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2278.408832] Lustre: Skipped 1 previous similar message [ 2282.826847] Lustre: server umount lustre-OST0000 complete [ 2288.615866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.639397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2288.647456] Lustre: Skipped 1 previous similar message [ 2288.667412] Lustre: Skipped 2 previous similar messages [ 2293.735872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2293.744852] Lustre: Skipped 1 previous similar message [ 2294.283629] Lustre: server umount lustre-MDT0000 complete [ 2298.583176] LustreError: 59912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772471621 with bad export cookie 17366160617369035631 [ 2298.620808] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2299.342655] Lustre: server umount lustre-MDT0001 complete [ 2311.218375] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 2314.609523] Key type lgssc unregistered [ 2314.925052] LNet: 63135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2314.937425] LNetError: 63135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2316.009929] LNet: Removed LNI 192.168.202.140@tcp [ 2317.026413] Key type .llcrypt unregistered [ 2317.028585] Key type ._llcrypt unregistered [ 2348.701708] Key type ._llcrypt registered [ 2348.704691] Key type .llcrypt registered [ 2348.886897] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 2350.386465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2350.410065] alg: No test for adler32 (adler32-zlib) [ 2351.714561] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 2352.228428] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 2354.071204] Key type lgssc registered [ 2355.815337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2367.287571] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 12:14:47 (1772471687) [ 2375.492096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2375.524048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.362931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.043711] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2387.940765] LustreError: 64827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.962127] LustreError: 64827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2393.067227] LustreError: 64826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2393.398451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.142627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2399.339654] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2406.346335] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2411.009964] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2419.510748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2420.130474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2422.203271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2427.371833] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2434.525166] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2445.268676] Lustre: Failing over lustre-MDT0000 [ 2445.280791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2445.290745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.320470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2447.507704] Lustre: server umount lustre-MDT0000 complete [ 2448.129313] LustreError: 64821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.871125] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.881222] Lustre: Skipped 1 previous similar message [ 2453.192768] LustreError: 66142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.205582] LustreError: 66142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2457.579096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.706937] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2457.842954] Lustre: *** cfs_fail_loc=135, val=0*** [ 2457.962962] LustreError: 67579:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000 failed (-2) [ 2457.971269] LustreError: 67579:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.202.140@tcp: cfg command failed: rc = -2 [ 2457.981207] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2457.981207] [ 2457.991934] LustreError: MGC192.168.202.140@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2458.012323] LustreError: 67567:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2458.026377] LustreError: 67567:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 2458.042233] LustreError: 67567:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 2458.048386] LustreError: 67567:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2458.085053] Lustre: server umount lustre-MDT0000 complete [ 2458.091637] LustreError: 67567:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2458.330927] LustreError: 66142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.353808] LustreError: 66142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2465.382027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2465.583147] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.085767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.183906] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2468.620944] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2471.053231] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2471.424770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2471.480680] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2471.533471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2481.635897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.673923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2486.774790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2486.796782] Lustre: Skipped 2 previous similar messages [ 2486.932870] Lustre: server umount lustre-OST0000 complete [ 2491.879429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2491.895269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2491.918075] Lustre: Skipped 1 previous similar message [ 2491.923773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2496.992529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2497.003680] Lustre: Skipped 2 previous similar messages [ 2497.750860] Lustre: server umount lustre-MDT0000 complete [ 2501.787214] LustreError: 64807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772471824 with bad export cookie 14811545833781217875 [ 2501.789707] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2501.794063] LustreError: 64807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2502.372490] Lustre: server umount lustre-MDT0001 complete [ 2516.523828] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 2520.093568] Key type lgssc unregistered [ 2520.570759] LNet: 69332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2520.593565] LNetError: 69332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2520.646811] LNet: Removed LNI 192.168.202.140@tcp [ 2522.306075] Key type .llcrypt unregistered [ 2522.307591] Key type ._llcrypt unregistered [ 2548.874988] Key type ._llcrypt registered [ 2548.876216] Key type .llcrypt registered [ 2549.072919] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 2550.512258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2550.550177] alg: No test for adler32 (adler32-zlib) [ 2551.687891] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 2551.905428] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 2553.543436] Key type lgssc registered [ 2555.377779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2566.292358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2566.321644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.068058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.613994] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2577.391145] LustreError: 70729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.411213] LustreError: 70729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2582.501812] LustreError: 70728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.547564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2585.162266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2590.695956] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2597.362676] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2601.664366] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2610.145132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2610.622753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2612.662265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2617.798795] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2623.742916] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2633.827891] Lustre: Failing over lustre-MDT0000 [ 2634.187519] Lustre: server umount lustre-MDT0000 complete [ 2635.233151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2635.241134] LustreError: 70724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2635.262559] Lustre: Skipped 2 previous similar messages [ 2635.283966] LustreError: 70724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2640.357040] LustreError: 71937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.379747] LustreError: 71937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2642.765137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.971618] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2643.168163] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2643.268590] LustreError: 73485:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000 failed (-9) [ 2643.274606] LustreError: 73485:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.202.140@tcp: cfg command failed: rc = -9 [ 2643.285226] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2643.285226] [ 2643.294523] LustreError: MGC192.168.202.140@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2643.304533] LustreError: 73473:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2643.312578] LustreError: 73473:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -9 [ 2643.326023] LustreError: 73473:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 2643.338647] LustreError: 73473:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2643.370602] Lustre: server umount lustre-MDT0000 complete [ 2643.376991] LustreError: 73473:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2645.474775] LustreError: 70729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2645.492397] LustreError: 70729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2651.438577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.544662] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2651.834763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.884855] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2652.363174] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2657.276526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2657.334503] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2657.420839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2657.969972] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2668.013413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2668.025835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.074767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2671.100843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2671.112853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.133396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2672.609303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2674.120601] Lustre: server umount lustre-OST0000 complete [ 2682.851153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2682.852773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2682.896915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2682.896924] Lustre: Skipped 1 previous similar message [ 2685.055142] Lustre: server umount lustre-MDT0000 complete [ 2687.977039] LustreError: 70728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.993850] LustreError: 70728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2689.631057] LustreError: 71447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772472012 with bad export cookie 16937560384731259071 [ 2689.634841] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2689.645418] LustreError: 71447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2690.163213] Lustre: server umount lustre-MDT0001 complete [ 2702.916188] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 2705.854134] Key type lgssc unregistered [ 2706.166839] LNet: 75234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2706.174261] LNetError: 75234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2706.186632] LNet: Removed LNI 192.168.202.140@tcp [ 2706.943380] Key type .llcrypt unregistered [ 2706.946145] Key type ._llcrypt unregistered [ 2722.752898] Key type ._llcrypt registered [ 2722.754521] Key type .llcrypt registered [ 2722.896750] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 2723.355299] Key type .llcrypt unregistered [ 2723.363272] Key type ._llcrypt unregistered [ 2749.592392] Key type ._llcrypt registered [ 2749.594745] Key type .llcrypt registered [ 2749.718448] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 2751.644500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2751.688231] alg: No test for adler32 (adler32-zlib) [ 2752.907610] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 2753.210115] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 2754.919204] Key type lgssc registered [ 2756.598107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2767.023589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2767.060392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.043680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2775.281985] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2778.596990] LustreError: 77106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2778.626843] LustreError: 77106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2783.713076] LustreError: 77105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2785.463738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2785.933453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2792.034538] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2798.291681] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2802.591961] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2810.621768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2810.866451] LustreError: 78700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2811.195661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2813.276021] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2818.838453] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2825.233883] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2839.012038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.021371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2839.034816] Lustre: Skipped 1 previous similar message [ 2839.035083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2839.074981] LustreError: Skipped 1 previous similar message [ 2842.467371] Lustre: server umount lustre-OST0000 complete [ 2847.200964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2847.204123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.235164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2847.239897] Lustre: Skipped 1 previous similar message [ 2849.258440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2849.265459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2849.271239] Lustre: Skipped 1 previous similar message [ 2853.228550] Lustre: server umount lustre-MDT0000 complete [ 2854.380100] LustreError: 78014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2854.412478] LustreError: 78014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2858.121734] LustreError: 77087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772472181 with bad export cookie 1649622473206879411 [ 2858.132014] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2858.137086] LustreError: 77087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2858.963286] Lustre: server umount lustre-MDT0001 complete [ 2874.180872] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 2878.446675] Key type lgssc unregistered [ 2878.752669] LNet: 80289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2878.759541] LNetError: 80289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2878.781044] LNet: Removed LNI 192.168.202.140@tcp [ 2879.731379] Key type .llcrypt unregistered [ 2879.732989] Key type ._llcrypt unregistered [ 2906.355735] Key type ._llcrypt registered [ 2906.359792] Key type .llcrypt registered [ 2906.531244] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 2908.441571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2908.468334] alg: No test for adler32 (adler32-zlib) [ 2909.586765] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 2909.927171] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 2911.655206] Key type lgssc registered [ 2913.448560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2923.765482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2923.784714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.716211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2931.940173] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2935.286857] LustreError: 81685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2935.311090] LustreError: 81685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2940.395142] LustreError: 81686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.302281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2942.895295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2948.724491] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2956.654351] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2961.189766] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2969.084227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2969.682570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2970.691269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2977.147933] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2983.859039] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2996.192273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2996.198319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2996.215861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2998.241980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2998.245858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.792814] Lustre: server umount lustre-OST0000 complete [ 3008.480423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3008.522258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3008.533580] Lustre: Skipped 2 previous similar messages [ 3013.610493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3013.622329] Lustre: Skipped 1 previous similar message [ 3018.721442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3018.733644] Lustre: Skipped 1 previous similar message [ 3019.236931] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3019.529798] Lustre: server umount lustre-MDT0000 complete [ 3023.777167] LustreError: 81665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772472346 with bad export cookie 12617115463134201858 [ 3023.782806] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3023.789221] LustreError: 81665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3024.119542] Lustre: server umount lustre-MDT0001 complete [ 3036.080630] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 3039.183690] Key type lgssc unregistered [ 3039.488786] LNet: 84880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3039.498213] LNetError: 84880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3040.550289] LNet: Removed LNI 192.168.202.140@tcp [ 3041.845948] Key type .llcrypt unregistered [ 3041.849914] Key type ._llcrypt unregistered [ 3067.211496] Key type ._llcrypt registered [ 3067.214704] Key type .llcrypt registered [ 3067.422590] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3068.826850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3068.977310] alg: No test for adler32 (adler32-zlib) [ 3070.251528] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 3070.642530] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 3072.295273] Key type lgssc registered [ 3074.037244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3082.559694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3082.595667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3084.374450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.283684] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3094.498901] LustreError: 86279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3094.524817] LustreError: 86279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3098.929602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.415044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3104.173400] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3111.172341] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3115.604786] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3123.871663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3124.325306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3130.383922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3132.098888] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3138.970251] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3155.952249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3155.957489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.002616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3156.011435] Lustre: Skipped 1 previous similar message [ 3161.061782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3161.076071] Lustre: Skipped 1 previous similar message [ 3161.626139] Lustre: server umount lustre-OST0000 complete [ 3171.298845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3171.303951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.307523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3171.307531] Lustre: Skipped 1 previous similar message [ 3171.339331] Lustre: Skipped 1 previous similar message [ 3172.645949] Lustre: server umount lustre-MDT0000 complete [ 3176.416300] LustreError: 86279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3176.433294] LustreError: 86279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3177.033545] LustreError: 86258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772472499 with bad export cookie 4690195569800150603 [ 3177.039462] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3177.062442] LustreError: 86258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3177.619322] Lustre: server umount lustre-MDT0001 complete [ 3193.202320] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 3196.379334] Key type lgssc unregistered [ 3196.706603] LNet: 89481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3196.717265] LNetError: 89481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3197.738813] LNet: Removed LNI 192.168.202.140@tcp [ 3198.951532] Key type .llcrypt unregistered [ 3198.953611] Key type ._llcrypt unregistered [ 3225.835871] Key type ._llcrypt registered [ 3225.841785] Key type .llcrypt registered [ 3226.007229] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3227.274949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3227.315549] alg: No test for adler32 (adler32-zlib) [ 3228.480469] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 3228.713658] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 3230.399691] Key type lgssc registered [ 3232.041483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3241.046579] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3241.075714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3256.671964] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3261.791495] LustreError: 90851:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.140@tcp: failed processing log, type 4: rc = -110 [ 3288.481970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3296.017729] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3302.682504] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3309.434217] Lustre: server umount lustre-OST0000 complete [ 3320.033978] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 12:30:40 (1772472640) [ 3327.034911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.389584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3331.826776] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3332.580757] LustreError: 92379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3332.596600] LustreError: 92379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3337.701839] LustreError: 92378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3342.823257] LustreError: 92379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.082222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.499276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3350.105801] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3356.714654] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3361.162593] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3368.614907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3369.060926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.149989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3374.589641] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3380.504183] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3398.114150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3398.126804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.152932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3399.138198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3399.142489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3399.152471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.650861] Lustre: server umount lustre-OST0000 complete [ 3409.376641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.379225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3409.410261] Lustre: Skipped 1 previous similar message [ 3409.430709] Lustre: Skipped 2 previous similar messages [ 3412.855121] Lustre: server umount lustre-MDT0000 complete [ 3414.507371] LustreError: 93264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3414.549893] LustreError: 93264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3417.372676] LustreError: 94001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772472740 with bad export cookie 2307191026942700563 [ 3417.375455] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3417.390036] LustreError: 94001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3417.684957] Lustre: server umount lustre-MDT0001 complete [ 3429.852571] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 3433.476754] Key type lgssc unregistered [ 3433.897894] LNet: 95722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3433.903041] LNetError: 95722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3433.925273] LNet: Removed LNI 192.168.202.140@tcp [ 3435.084164] Key type .llcrypt unregistered [ 3435.086328] Key type ._llcrypt unregistered [ 3462.189321] Key type ._llcrypt registered [ 3462.193651] Key type .llcrypt registered [ 3462.414648] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3464.372059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3464.406479] alg: No test for adler32 (adler32-zlib) [ 3465.647359] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 3465.845570] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 3467.575894] Key type lgssc registered [ 3469.132785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3479.257564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3479.323196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3481.077423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.661297] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3490.308781] LustreError: 97126:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3490.330603] LustreError: 97126:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3495.392167] LustreError: 97125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3498.037487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3498.673382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3504.056562] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3510.184420] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3514.507767] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3522.864700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3523.384144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.422856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3530.881584] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3537.830134] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3550.176597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3550.191842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.209360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.204111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.206261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.219652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3555.728044] Lustre: server umount lustre-OST0000 complete [ 3561.441209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3561.457804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3561.460543] Lustre: Skipped 1 previous similar message [ 3566.434891] Lustre: server umount lustre-MDT0000 complete [ 3566.568529] LustreError: 97125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.592941] LustreError: 97125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3571.184414] LustreError: 97106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772472894 with bad export cookie 11479567407149642326 [ 3571.191437] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3571.199729] LustreError: 97106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3571.530532] Lustre: server umount lustre-MDT0001 complete [ 3583.136945] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 3585.951151] Key type lgssc unregistered [ 3586.191686] LNet: 100305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3586.202175] LNetError: 100305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3587.258587] LNet: Removed LNI 192.168.202.140@tcp [ 3588.400251] Key type .llcrypt unregistered [ 3588.405273] Key type ._llcrypt unregistered [ 3610.283405] Key type ._llcrypt registered [ 3610.289572] Key type .llcrypt registered [ 3610.468431] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3612.133936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3612.154054] alg: No test for adler32 (adler32-zlib) [ 3613.224780] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 3613.525316] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 3615.335281] Key type lgssc registered [ 3616.655556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3625.645614] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3625.672378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3641.375343] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3646.495295] LustreError: 101536:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.140@tcp: failed processing log, type 4: rc = -110 [ 3673.249128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3680.837797] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3687.508471] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3695.103104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3695.349643] LustreError: MGC192.168.202.140@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3695.358549] LustreError: 102447:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3695.364963] LustreError: 102447:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 3695.377871] LustreError: 102447:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 3695.460834] Lustre: server umount lustre-MDT0000 complete [ 3695.464486] LustreError: 102447:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3717.089673] Lustre: 100872:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772473023/real 1772473023] req@0000000022fb8c4e x1858572581342592/t0(0) o400->MGC192.168.202.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772473039 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3717.122703] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3717.147179] LustreError: 101548:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3737.676688] Lustre: server umount lustre-OST0000 complete [ 3747.713228] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 3759.017629] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3801.723787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3802.029796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3802.112769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3802.275452] Lustre: lustre-MDT0000: new disk, initializing [ 3802.460783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3802.492700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3808.906632] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3822.638924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3822.823478] Lustre: 105208:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3822.940686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3822.956189] Lustre: Skipped 1 previous similar message [ 3823.122810] Lustre: lustre-MDT0001: new disk, initializing [ 3823.302347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3823.406368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3823.430204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3829.229203] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3840.657772] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3844.434367] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3854.477318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3855.115360] Lustre: lustre-OST0000: new disk, initializing [ 3855.132913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3855.337796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.718083] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3863.167954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3863.180479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3863.328306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3872.374802] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3878.801168] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3879.180611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3885.613270] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3885.905349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3889.136023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3889.140721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3889.151928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3889.162724] Lustre: Skipped 1 previous similar message [ 3894.261320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3894.270176] Lustre: Skipped 2 previous similar messages [ 3894.748648] Lustre: server umount lustre-OST0000 complete [ 3899.362308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3899.384597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3899.400950] Lustre: Skipped 1 previous similar message [ 3904.483711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3904.496891] Lustre: Skipped 1 previous similar message [ 3905.776829] Lustre: server umount lustre-MDT0000 complete [ 3910.784835] LustreError: 105200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473233 with bad export cookie 14921998886320422723 [ 3910.814460] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3911.214757] Lustre: server umount lustre-MDT0001 complete [ 3922.120861] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 12:40:42 (1772473242) [ 3926.616648] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3943.671437] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 3957.214668] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 4001.382388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.673167] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4001.728828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4001.797987] Lustre: lustre-MDT0000: new disk, initializing [ 4001.915053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4001.944888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4008.780476] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4024.787866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4025.144802] Lustre: 111674:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4025.254850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4025.258100] Lustre: Skipped 1 previous similar message [ 4025.328891] Lustre: lustre-MDT0001: new disk, initializing [ 4025.450991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4025.510950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4025.528901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4030.736195] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4040.048285] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4044.986769] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4053.042420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4053.602209] Lustre: lustre-OST0000: new disk, initializing [ 4053.615221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4053.742042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4060.801594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4060.818636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4061.030590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4061.341388] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4069.642816] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4074.339679] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4074.585791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4080.411659] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4080.716380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4086.755045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4086.758808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4086.759408] LustreError: Skipped 2 previous similar messages [ 4086.769012] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4086.800236] Lustre: Skipped 2 previous similar messages [ 4086.827526] Lustre: Skipped 1 previous similar message [ 4089.408810] Lustre: server umount lustre-OST0000 complete [ 4096.992646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4096.995623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.008323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4097.039084] Lustre: Skipped 1 previous similar message [ 4097.059792] Lustre: Skipped 1 previous similar message [ 4099.791691] Lustre: server umount lustre-MDT0000 complete [ 4102.112409] LustreError: 111682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4102.130692] LustreError: 111682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4103.945597] LustreError: 111666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473426 with bad export cookie 14921998886320423605 [ 4103.954216] LustreError: 111666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4103.958881] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4104.517437] Lustre: server umount lustre-MDT0001 complete [ 4114.596321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4115.361350] LustreError: 115490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4115.453245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4120.152271] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4120.545171] LustreError: 115491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4125.667087] LustreError: 115490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4129.877515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4135.476987] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4142.053649] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4146.314431] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4153.019657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4159.440565] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4165.484346] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4171.912801] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4176.355160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4176.369531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4176.388871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4182.551213] Lustre: server umount lustre-OST0000 complete [ 4186.593633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4186.600503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4186.603056] LustreError: Skipped 1 previous similar message [ 4186.622748] Lustre: Skipped 2 previous similar messages [ 4192.862219] Lustre: server umount lustre-MDT0000 complete [ 4196.643183] LustreError: 115470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473519 with bad export cookie 14921998886320424480 [ 4196.654846] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4196.657307] LustreError: 115470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4196.965727] Lustre: server umount lustre-MDT0001 complete [ 4207.548721] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 4209.999796] Key type lgssc unregistered [ 4210.271580] LNet: 118769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4210.278385] LNetError: 118769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4210.306416] LNet: Removed LNI 192.168.202.140@tcp [ 4211.329781] Key type .llcrypt unregistered [ 4211.331993] Key type ._llcrypt unregistered [ 4227.976404] Key type ._llcrypt registered [ 4227.978797] Key type .llcrypt registered [ 4228.121290] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 4242.096735] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 4242.973257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4242.984855] alg: No test for adler32 (adler32-zlib) [ 4244.052250] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 4244.255726] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 4245.911173] Key type lgssc registered [ 4246.739954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4277.158993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4277.207363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.582722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4278.611743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4278.712922] Lustre: lustre-MDT0000: new disk, initializing [ 4278.824320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4278.875329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4283.266764] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4292.793826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4292.928596] Lustre: 121889:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4292.989995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4292.996349] Lustre: Skipped 1 previous similar message [ 4293.063433] Lustre: lustre-MDT0001: new disk, initializing [ 4293.154613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4293.204249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4293.220418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4297.437064] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4304.300463] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4306.825273] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4312.626630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4312.855534] Lustre: lustre-OST0000: new disk, initializing [ 4312.859401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4312.920878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4318.345140] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4322.414684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4322.437970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4322.549336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4325.086237] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4328.939629] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4329.131943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4333.121940] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4333.298807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4338.154287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4338.161060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.166114] LustreError: Skipped 1 previous similar message [ 4338.168204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4338.191569] Lustre: Skipped 1 previous similar message [ 4341.167510] Lustre: server umount lustre-OST0000 complete [ 4348.391254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.393829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4348.408395] Lustre: Skipped 1 previous similar message [ 4348.421706] Lustre: Skipped 2 previous similar messages [ 4350.682402] Lustre: server umount lustre-MDT0000 complete [ 4353.708529] LustreError: 121880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473676 with bad export cookie 1020709075345660247 [ 4353.720085] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4353.982654] Lustre: server umount lustre-MDT0001 complete [ 4360.391773] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 12:48:01 (1772473681) [ 4365.704214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4366.292992] LustreError: 125889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4366.320796] LustreError: 125889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4366.395956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4370.153678] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4371.444658] LustreError: 125890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4376.549882] LustreError: 125889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4377.718567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4378.000934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4382.106936] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4387.071739] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4390.394883] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4393.441971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4393.450801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4393.466433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4396.512515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4396.513023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4396.528980] Lustre: Skipped 1 previous similar message [ 4398.294565] Lustre: server umount lustre-MDT0000 complete [ 4401.394677] LustreError: 125870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473724 with bad export cookie 1020709075345661122 [ 4401.403513] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4401.406691] LustreError: 125870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4401.720896] Lustre: server umount lustre-MDT0001 complete [ 4408.699738] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 12:48:49 (1772473729) [ 4414.094550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4429.727590] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4434.847569] LustreError: 128075:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.140@tcp: failed processing log, type 4: rc = -110 [ 4460.577024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4465.095151] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4468.541757] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4470.139247] Lustre: server umount lustre-OST0000 complete [ 4476.341082] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 12:49:57 (1772473797) [ 4481.968272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4482.372652] LustreError: 129406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.440589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4485.955498] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4487.652218] LustreError: 129407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4493.150744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4497.052955] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4501.928526] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4504.670198] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4509.923651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4514.418824] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4518.531422] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4534.240054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4534.245580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4534.252620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4539.748245] Lustre: server umount lustre-MDT0000 complete [ 4542.661644] LustreError: 129388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473865 with bad export cookie 1020709075345661794 [ 4542.672863] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4542.915403] Lustre: server umount lustre-MDT0001 complete [ 4555.986587] Lustre: server umount lustre-OST0000 complete [ 4562.270951] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 12:51:23 (1772473883) [ 4566.783381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.125128] LustreError: 132745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4567.148395] LustreError: 132745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4567.195514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.200373] Lustre: Skipped 2 previous similar messages [ 4570.434670] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4576.863053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4579.921337] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4583.557116] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4586.041931] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4590.417053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4593.708427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4594.528461] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4598.220661] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4600.586926] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4600.702138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4603.525539] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4603.693210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4606.958950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4606.961749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4606.967333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4606.979656] Lustre: Skipped 3 previous similar messages [ 4606.984109] Lustre: Skipped 3 previous similar messages [ 4611.144185] Lustre: server umount lustre-OST0000 complete [ 4620.100746] Lustre: server umount lustre-MDT0000 complete [ 4622.308419] LustreError: 132740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4622.316683] LustreError: 132740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4622.808433] LustreError: 132727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473945 with bad export cookie 1020709075345662865 [ 4622.815338] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4622.818062] LustreError: 132727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4623.018987] Lustre: server umount lustre-MDT0001 complete [ 4628.483745] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 12:52:30 (1772473950) [ 4632.383969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4647.839783] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4652.959617] LustreError: 136530:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.140@tcp: failed processing log, type 4: rc = -110 [ 4678.623593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4678.628797] Lustre: Skipped 2 previous similar messages [ 4683.227295] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4687.134566] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4691.563371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4691.853260] LustreError: 136555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4691.943235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4695.028666] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4700.536944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4703.491239] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4707.340891] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4709.856408] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4712.781381] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4712.955114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4716.164239] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4716.350687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4721.120364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4721.123246] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4721.126952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4721.126958] Lustre: Skipped 3 previous similar messages [ 4721.142388] Lustre: Skipped 4 previous similar messages [ 4723.943367] Lustre: server umount lustre-MDT0000 complete [ 4726.221934] LustreError: 136536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474049 with bad export cookie 1020709075345663887 [ 4726.227152] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4726.227472] LustreError: 136536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4726.400510] Lustre: server umount lustre-MDT0001 complete [ 4734.942559] Lustre: server umount lustre-OST0000 complete [ 4740.254183] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 12:54:21 (1772474061) [ 4744.266792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4759.775666] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4764.895444] LustreError: 140349:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.140@tcp: failed processing log, type 4: rc = -110 [ 4794.808053] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4798.527614] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4803.571275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4803.883680] LustreError: 140374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4803.891766] LustreError: 140374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4804.016840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4807.099809] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4812.242278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4812.451559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4812.458790] Lustre: Skipped 4 previous similar messages [ 4814.664335] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4818.057230] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4820.295157] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4824.802803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4824.923967] Lustre: lustre-OST0001: new disk, initializing [ 4824.927172] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4825.694431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4825.702799] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4825.784434] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4829.064868] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4834.380892] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4836.927902] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4837.060801] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4839.725481] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4839.854754] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4841.442991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4841.451178] Lustre: Skipped 2 previous similar messages [ 4841.453508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4841.458878] Lustre: Skipped 4 previous similar messages [ 4847.314462] Lustre: server umount lustre-OST0000 complete [ 4851.679917] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4863.967489] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4864.097985] Lustre: server umount lustre-OST0001 complete [ 4872.847621] Lustre: server umount lustre-MDT0000 complete [ 4875.243888] LustreError: 140355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474198 with bad export cookie 1020709075345664839 [ 4875.247796] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4875.252333] LustreError: 140355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4875.477751] Lustre: server umount lustre-MDT0001 complete [ 4892.012822] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 12:56:53 (1772474213) [ 4893.219549] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4894.287735] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4895.326230] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 12:56:56 (1772474216) [ 4899.841190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4900.089484] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4900.121058] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4902.996317] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4908.314666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4908.381607] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4908.413865] Lustre: 146294:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4911.591995] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4915.150289] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4917.791921] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4922.272197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4922.408824] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4923.540606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4926.767406] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4930.778545] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4933.953142] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4934.109358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4937.418326] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4937.585198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4939.237146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4939.238854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4939.246426] Lustre: Skipped 4 previous similar messages [ 4939.252457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4939.252462] Lustre: Skipped 12 previous similar messages [ 4953.056065] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4953.154256] Lustre: server umount lustre-OST0000 complete [ 4970.822770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4974.880297] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4978.221646] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4979.302268] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.310753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.311417] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4979.321713] Lustre: Skipped 1 previous similar message [ 4981.289607] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4981.442329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4983.697402] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4983.802639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4985.878862] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8ff5c413c000.ost_server_uuid 50 [ 4986.666864] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8ff5c413c000.ost_server_uuid in FULL state after 0 sec [ 4986.848689] LustreError: 146302:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.40@tcp) returned error from blocking AST (req@00000000ee97418b x1858573243216384 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000f08c758c/0xe2a497efefef4a1 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.40@tcp remote: 0x4d542921ddc2461e expref: 5 pid: 146301 timeout: 5086 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4986.871479] LustreError: lustre-MDT0000: A client on nid 192.168.202.40@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4986.876078] LustreError: 146291:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.40@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000f08c758c/0xe2a497efefef4a1 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.40@tcp remote: 0x4d542921ddc2461e expref: 6 pid: 146301 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4989.924552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4995.048198] Lustre: server umount lustre-OST0000 complete [ 5003.866647] Lustre: server umount lustre-MDT0000 complete [ 5005.280936] LustreError: 149185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5005.298611] LustreError: 149185:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 5006.685997] LustreError: 146286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474329 with bad export cookie 1020709075345666022 [ 5006.689436] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5006.701843] LustreError: 146286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5021.151308] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5021.342830] Lustre: server umount lustre-MDT0001 complete [ 5028.873388] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 5030.780606] Key type lgssc unregistered [ 5031.028659] LNet: 151773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5031.039939] LNetError: 151773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5031.057755] LNet: Removed LNI 192.168.202.140@tcp [ 5031.618497] Key type .llcrypt unregistered [ 5031.620294] Key type ._llcrypt unregistered [ 5047.319753] Key type ._llcrypt registered [ 5047.320884] Key type .llcrypt registered [ 5047.377881] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5047.979402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5048.052884] alg: No test for adler32 (adler32-zlib) [ 5049.069232] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 5049.210529] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 5050.839694] Key type lgssc registered [ 5051.557688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5056.507374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5056.523909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.924672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.918518] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5066.549487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.792790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5069.655726] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5073.669808] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5076.438092] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5081.358544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5081.642321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5082.679282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5085.416657] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5088.973510] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5098.147406] Lustre: Failing over lustre-MDT0000 [ 5098.325203] Lustre: server umount lustre-MDT0000 complete [ 5102.049586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5102.051680] LustreError: 153559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5102.068852] Lustre: Skipped 1 previous similar message [ 5102.089310] LustreError: 153559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5107.168710] LustreError: 153147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5107.183156] LustreError: 153147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5109.841410] LustreError: 153127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.40@tcp arrived at 1772474432 with bad export cookie 15699945274910158353 [ 5109.855447] LustreError: 153127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5112.291548] LustreError: 153859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5112.304188] LustreError: 153859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5115.345675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5115.433650] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5115.588295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5115.617958] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5118.705027] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5120.271774] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5120.279779] Lustre: lustre-MDT0000: Denying connection for new client c27b7e90-7d49-4f17-88f2-700cb1332b47 (at 192.168.202.40@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 5121.004195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5127.647919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5127.654181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5127.665641] Lustre: Skipped 1 previous similar message [ 5127.670946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5131.237515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5131.243125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5131.251651] Lustre: Skipped 1 previous similar message [ 5133.782930] Lustre: server umount lustre-OST0000 complete [ 5136.249688] LustreError: 156521:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5136.254798] Lustre: 155886:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5136.262356] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5136.278993] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5136.292198] LustreError: 155886:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000beac38d5 x1858574087133312/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5136.306887] LustreError: 155886:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5136.313354] LustreError: 155886:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5136.319650] Lustre: lustre-MDT0000: Recovery over after 0:16, of 2 clients 0 recovered and 2 were evicted. [ 5136.352741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.360548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5136.575570] Lustre: server umount lustre-MDT0000 complete [ 5139.388075] LustreError: 153126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474462 with bad export cookie 15699945274910158458 [ 5139.396547] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5139.398844] LustreError: 153126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5139.416395] LustreError: 153146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5145.715890] Lustre: server umount lustre-MDT0001 complete [ 5151.863921] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 5153.648322] Key type lgssc unregistered [ 5153.831624] LNet: 157266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5153.838617] LNetError: 157266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5154.855833] LNet: Removed LNI 192.168.202.140@tcp [ 5155.355943] Key type .llcrypt unregistered [ 5155.360441] Key type ._llcrypt unregistered [ 5171.591563] Key type ._llcrypt registered [ 5171.593207] Key type .llcrypt registered [ 5171.664920] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5172.400827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5172.412336] alg: No test for adler32 (adler32-zlib) [ 5173.390444] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 5173.561979] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 5175.199183] Key type lgssc registered [ 5175.984956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5181.146571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5181.156808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5182.471668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5185.878904] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5191.855966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5192.204538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5194.950086] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5198.911326] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5201.288373] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5206.219934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5206.434831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5208.498986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5210.430455] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5214.033916] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5218.786609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5218.791665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5218.804939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5222.881568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5222.885365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5222.887477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5222.906156] Lustre: Skipped 1 previous similar message [ 5224.483446] Lustre: server umount lustre-OST0000 complete [ 5228.005082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5228.005260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5228.016693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5231.011824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5231.018635] Lustre: Skipped 2 previous similar messages [ 5232.815035] Lustre: server umount lustre-MDT0000 complete [ 5234.997676] LustreError: 160259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474557 with bad export cookie 546020221609911173 [ 5234.998701] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5235.003105] LustreError: 160259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5235.215780] Lustre: server umount lustre-MDT0001 complete [ 5241.591296] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 5243.367423] Key type lgssc unregistered [ 5243.603082] LNet: 161820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5243.612616] LNetError: 161820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5243.623892] LNet: Removed LNI 192.168.202.140@tcp [ 5244.292423] Key type .llcrypt unregistered [ 5244.293920] Key type ._llcrypt unregistered [ 5265.290508] Key type ._llcrypt registered [ 5265.298523] Key type .llcrypt registered [ 5265.400633] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5266.271048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5266.288809] alg: No test for adler32 (adler32-zlib) [ 5267.261726] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 5267.415534] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 5269.063390] Key type lgssc registered [ 5269.948841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5275.344992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5275.362492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5276.741492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5279.810149] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5286.148672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5286.467564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5289.785593] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5293.501534] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5296.158070] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5300.463307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5300.701847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5302.768934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5304.931755] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5308.772537] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5318.113211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5318.116969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5318.124598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5322.209686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5322.213732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5324.185049] Lustre: server umount lustre-OST0000 complete [ 5327.333866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5327.335333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5327.337247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5327.337254] Lustre: Skipped 1 previous similar message [ 5327.363991] Lustre: Skipped 1 previous similar message [ 5332.451158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5332.456025] Lustre: Skipped 2 previous similar messages [ 5332.863239] Lustre: server umount lustre-MDT0000 complete [ 5335.081344] LustreError: 164201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474657 with bad export cookie 5374704396392925450 [ 5335.083951] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5335.094831] LustreError: 164201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5335.453202] Lustre: server umount lustre-MDT0001 complete [ 5342.368437] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 5344.540959] Key type lgssc unregistered [ 5344.736196] LNet: 166661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5344.741064] LNetError: 166661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5344.751660] LNet: Removed LNI 192.168.202.140@tcp [ 5345.344383] Key type .llcrypt unregistered [ 5345.345650] Key type ._llcrypt unregistered [ 5360.283321] Key type ._llcrypt registered [ 5360.285584] Key type .llcrypt registered [ 5360.396816] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5361.074869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5361.173516] alg: No test for adler32 (adler32-zlib) [ 5362.133573] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 5362.266218] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 5363.887190] Key type lgssc registered [ 5364.582259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5371.432628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5371.446143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5372.658237] Lustre: *** cfs_fail_loc=135, val=0*** [ 5372.716221] LustreError: 168146:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000 failed (-2) [ 5372.722273] LustreError: 168146:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.202.140@tcp: cfg command failed: rc = -2 [ 5372.727609] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5372.727609] [ 5372.730598] LustreError: MGC192.168.202.140@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5372.737719] LustreError: 168106:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5372.742368] LustreError: 168106:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 5372.745547] LustreError: 168106:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 5372.749275] LustreError: 168106:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 5372.794179] Lustre: server umount lustre-MDT0000 complete [ 5372.795910] LustreError: 168106:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5377.296881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5377.654311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5380.155274] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5386.115788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5386.336124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5389.096300] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5392.704352] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5395.040220] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5396.450227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5396.456394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5396.473753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5397.473951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5397.474608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5402.594167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5402.602268] Lustre: Skipped 2 previous similar messages [ 5402.731632] Lustre: server umount lustre-MDT0000 complete [ 5405.396894] LustreError: 168498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474728 with bad export cookie 2601884811642342253 [ 5405.404117] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5405.599752] Lustre: server umount lustre-MDT0001 complete [ 5412.795803] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 5414.828978] Key type lgssc unregistered [ 5415.042253] LNet: 170591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5415.050505] LNetError: 170591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5415.070995] LNet: Removed LNI 192.168.202.140@tcp [ 5415.647419] Key type .llcrypt unregistered [ 5415.648755] Key type ._llcrypt unregistered [ 5429.971589] Key type ._llcrypt registered [ 5429.973033] Key type .llcrypt registered [ 5430.078404] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 5430.267487] Key type .llcrypt unregistered [ 5430.270118] Key type ._llcrypt unregistered [ 5440.248204] Key type ._llcrypt registered [ 5440.249441] Key type .llcrypt registered [ 5440.304786] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5440.777354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5440.815105] alg: No test for adler32 (adler32-zlib) [ 5441.811904] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 5441.959667] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 5443.599594] Key type lgssc registered [ 5444.259725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5448.892847] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5448.916744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5464.479656] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5469.599449] LustreError: 172669:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.140@tcp: failed processing log, type 4: rc = -110 [ 5496.223238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5499.476134] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5502.460112] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5506.439574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5506.769365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5506.798988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5509.555781] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5515.037780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.392634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5518.097106] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5521.399232] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5523.584298] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5525.338746] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5532.642110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5532.648648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5532.656064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5535.713325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5535.716314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5535.721423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5537.685802] Lustre: server umount lustre-OST0000 complete [ 5540.832183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5540.838095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5540.843813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5540.846120] Lustre: Skipped 2 previous similar messages [ 5545.929458] Lustre: server umount lustre-MDT0000 complete [ 5546.464934] LustreError: 173969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5546.470686] LustreError: 173969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5548.184962] LustreError: 172677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474871 with bad export cookie 4975155826866276953 [ 5548.185585] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5548.191331] LustreError: 172677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5548.397335] Lustre: server umount lustre-MDT0001 complete [ 5554.788187] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 5556.553834] Key type lgssc unregistered [ 5556.752650] LNet: 176230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5556.764379] LNetError: 176230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5556.776453] LNet: Removed LNI 192.168.202.140@tcp [ 5557.350675] Key type .llcrypt unregistered [ 5557.352644] Key type ._llcrypt unregistered [ 5572.105825] Key type ._llcrypt registered [ 5572.106852] Key type .llcrypt registered [ 5572.150332] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5572.944240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5572.955862] alg: No test for adler32 (adler32-zlib) [ 5573.917204] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 5574.059918] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 5575.671233] Key type lgssc registered [ 5576.219213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5580.922937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5580.936603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5582.280498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.520704] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5589.871139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5590.165833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5592.707481] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5595.896652] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5598.192351] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5602.259089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5602.466064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5605.753392] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5607.597910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5609.020780] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5613.410378] Lustre: Failing over lustre-MDT0000 [ 5613.613821] Lustre: server umount lustre-MDT0000 complete [ 5614.443537] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5614.453124] Lustre: Skipped 2 previous similar messages [ 5614.457526] LustreError: 177601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5614.482041] LustreError: 177601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5616.863863] LustreError: 177600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5619.681850] LustreError: 177601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5619.697236] LustreError: 177601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5621.960389] LustreError: 177600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5627.085197] LustreError: 177602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5627.101924] LustreError: 177602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5629.770454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5629.861683] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5629.921302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 5630.052709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5630.092144] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5632.200037] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5632.595652] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5635.241535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5635.556627] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5635.569167] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5635.598823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5636.533470] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5645.797703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5645.798396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5645.800678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5645.800685] Lustre: Skipped 2 previous similar messages [ 5645.821533] Lustre: Skipped 1 previous similar message [ 5649.336509] Lustre: server umount lustre-OST0000 complete [ 5656.033868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.034085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5656.039137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5656.039146] Lustre: Skipped 1 previous similar message [ 5657.457771] Lustre: server umount lustre-MDT0000 complete [ 5659.655485] LustreError: 180819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474982 with bad export cookie 5224339801928855225 [ 5659.658026] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5659.664791] LustreError: 180819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5659.870560] Lustre: server umount lustre-MDT0001 complete [ 5665.537210] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 5667.072792] Key type lgssc unregistered [ 5667.271637] LNet: 181969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5667.276895] LNetError: 181969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5667.284966] LNet: Removed LNI 192.168.202.140@tcp [ 5667.759390] Key type .llcrypt unregistered [ 5667.760551] Key type ._llcrypt unregistered [ 5681.742856] Key type ._llcrypt registered [ 5681.744859] Key type .llcrypt registered [ 5681.794342] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5682.193148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5682.205785] alg: No test for adler32 (adler32-zlib) [ 5683.097136] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 5683.236453] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 5684.839212] Key type lgssc registered [ 5685.375813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5689.988664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5690.000054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.260769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5694.273262] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5699.065439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.329523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5701.614361] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5705.022708] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5707.152626] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5711.220493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5711.449685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5714.660690] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5716.581933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5717.820741] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5720.984553] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5729.145575] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5739.121166] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5752.802495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5752.806301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.818578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5755.872130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5755.874823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5755.882131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5755.895268] Lustre: Skipped 1 previous similar message [ 5757.017705] Lustre: server umount lustre-OST0000 complete [ 5760.991900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5760.993707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5761.004919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5761.010577] Lustre: Skipped 1 previous similar message [ 5761.027735] Lustre: Skipped 1 previous similar message [ 5765.504206] Lustre: server umount lustre-MDT0000 complete [ 5766.112134] LustreError: 183342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5766.124650] LustreError: 183342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5767.511681] LustreError: 185536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475090 with bad export cookie 8488087545735531516 [ 5767.516962] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5767.519715] LustreError: 185536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5767.721666] Lustre: server umount lustre-MDT0001 complete [ 5773.348694] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 5774.829839] Key type lgssc unregistered [ 5774.982656] LNet: 186705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5774.986905] LNetError: 186705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5776.043655] LNet: Removed LNI 192.168.202.140@tcp [ 5776.596569] Key type .llcrypt unregistered [ 5776.597845] Key type ._llcrypt unregistered [ 5789.304972] Key type ._llcrypt registered [ 5789.306545] Key type .llcrypt registered [ 5789.361984] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5789.870156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5789.901782] alg: No test for adler32 (adler32-zlib) [ 5790.825378] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 5791.015422] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 5792.647415] Key type lgssc registered [ 5793.184417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5797.319870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5797.332571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5798.565306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5800.654660] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5805.248699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5805.500199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5807.623569] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5810.620344] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5812.746993] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5815.941613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5816.121758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5818.856454] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5821.610522] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5823.271366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5831.033380] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5840.195931] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5845.351035] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5854.068033] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5864.416053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5864.419916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5864.426942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5866.466765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5866.469544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5866.470131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5866.476785] Lustre: Skipped 1 previous similar message [ 5871.585246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5871.591779] Lustre: Skipped 1 previous similar message [ 5876.707779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5876.717273] Lustre: Skipped 1 previous similar message [ 5877.215191] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5877.276152] Lustre: server umount lustre-OST0000 complete [ 5881.824489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5881.830203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5881.836861] Lustre: Skipped 1 previous similar message [ 5881.847302] Lustre: Skipped 1 previous similar message [ 5892.065263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5892.068452] Lustre: Skipped 3 previous similar messages [ 5893.087125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5893.160917] Lustre: server umount lustre-MDT0000 complete [ 5894.899830] LustreError: 188059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475217 with bad export cookie 11185586821123001718 [ 5894.903755] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5894.907484] LustreError: 188059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5895.070769] Lustre: server umount lustre-MDT0001 complete [ 5899.572932] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 5901.058750] Key type lgssc unregistered [ 5901.229527] LNet: 192987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5901.232449] LNetError: 192987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5901.241564] LNet: Removed LNI 192.168.202.140@tcp [ 5901.641593] Key type .llcrypt unregistered [ 5901.642777] Key type ._llcrypt unregistered [ 5913.265380] Key type ._llcrypt registered [ 5913.267142] Key type .llcrypt registered [ 5913.318974] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5913.808422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5913.828779] alg: No test for adler32 (adler32-zlib) [ 5914.750238] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 5914.858814] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 5916.479310] Key type lgssc registered [ 5917.003594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5920.661311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5920.671595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5921.923448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5923.913267] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5928.087805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.271135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5929.951933] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5932.271540] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5933.913384] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5936.985035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5937.152735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5939.431593] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5941.690965] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5946.410716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5953.503956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5953.506816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5953.512315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5956.576790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5956.578507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5956.578979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5956.588201] Lustre: Skipped 1 previous similar message [ 5961.697162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5961.699345] Lustre: Skipped 1 previous similar message [ 5966.303410] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5966.385237] Lustre: server umount lustre-OST0000 complete [ 5969.377284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5969.381053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5969.386113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5971.936976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5974.210740] Lustre: server umount lustre-MDT0000 complete [ 5975.948373] LustreError: 194342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475298 with bad export cookie 4119464695328211956 [ 5975.952024] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5975.953082] LustreError: 194342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5976.083226] Lustre: server umount lustre-MDT0001 complete [ 5980.682731] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 5982.088772] Key type lgssc unregistered [ 5982.261582] LNet: 197689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5982.265578] LNetError: 197689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5982.275523] LNet: Removed LNI 192.168.202.140@tcp [ 5982.680976] Key type .llcrypt unregistered [ 5982.683375] Key type ._llcrypt unregistered [ 5991.568576] Key type ._llcrypt registered [ 5991.569987] Key type .llcrypt registered [ 5991.614511] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5991.993969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5992.026088] alg: No test for adler32 (adler32-zlib) [ 5992.899619] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 5993.012764] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 5994.615620] Key type lgssc registered [ 5995.204812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5998.662843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5998.669777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.930954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6001.608898] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6005.435349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.632880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6007.346630] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6009.770906] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6011.390514] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6014.402166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6014.575910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6017.043133] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6019.517330] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6024.808245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6030.304754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6030.310388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6030.317684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6030.320143] Lustre: Skipped 1 previous similar message [ 6030.332095] Lustre: Skipped 1 previous similar message [ 6031.799118] Lustre: server umount lustre-OST0000 complete [ 6035.428290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6035.435283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6035.438508] Lustre: Skipped 1 previous similar message [ 6035.830120] Lustre: server umount lustre-MDT0000 complete [ 6037.486598] LustreError: 198895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475360 with bad export cookie 7626694852578033428 [ 6037.492352] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6037.493166] LustreError: 198895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6037.668267] Lustre: server umount lustre-MDT0001 complete [ 6042.090289] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 6043.299555] Key type lgssc unregistered [ 6043.441370] LNet: 202238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6043.444387] LNetError: 202238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6043.453308] LNet: Removed LNI 192.168.202.140@tcp [ 6043.783254] Key type .llcrypt unregistered [ 6043.784307] Key type ._llcrypt unregistered [ 6052.066207] Key type ._llcrypt registered [ 6052.067567] Key type .llcrypt registered [ 6052.121477] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 6052.568472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6052.576135] alg: No test for adler32 (adler32-zlib) [ 6053.454848] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 6053.572758] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 6055.175192] Key type lgssc registered [ 6055.669907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6058.992768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6059.000221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6060.236630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6062.008489] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6065.774527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6065.974477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6067.612987] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6069.847420] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6071.383628] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6074.169899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6074.361586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6076.817617] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6079.313822] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6085.609732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6093.433023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6093.471525] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6093.573813] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6096.133706] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6098.223209] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6098.839779] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6109.474314] Lustre: Permanently deactivating lustre-OST0001 [ 6109.480323] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6110.471044] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6120.599623] Lustre: Permanently reactivating lustre-OST0001 [ 6120.607440] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6120.614475] Lustre: Skipped 2 previous similar messages [ 6129.154275] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6129.164064] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6129.167805] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6129.173139] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6129.193324] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6133.219663] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6133.221307] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6133.225358] Lustre: Skipped 2 previous similar messages [ 6133.229324] Lustre: Skipped 1 previous similar message [ 6136.127996] Lustre: server umount lustre-OST0001 complete [ 6138.338024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6138.339777] LustreError: 206580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6138.341147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6138.345413] Lustre: Skipped 1 previous similar message [ 6138.353314] LustreError: 206580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6143.460507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6143.461960] LustreError: 206580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.462892] Lustre: Skipped 2 previous similar messages [ 6143.471817] LustreError: 206580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6148.576981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6148.577058] LustreError: 206592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6148.580656] Lustre: Skipped 1 previous similar message [ 6148.588811] LustreError: 206592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6152.159124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6152.261374] Lustre: server umount lustre-OST0000 complete [ 6158.303842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6158.306516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6158.313454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6160.162217] Lustre: server umount lustre-MDT0000 complete [ 6161.592283] LustreError: 204180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475484 with bad export cookie 7823358001241254591 [ 6161.594566] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6161.596159] LustreError: 204180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6161.723603] Lustre: server umount lustre-MDT0001 complete [ 6166.153651] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 6167.301355] Key type lgssc unregistered [ 6167.426446] LNet: 208056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6167.430912] LNetError: 208056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6167.445396] LNet: Removed LNI 192.168.202.140@tcp [ 6167.797948] Key type .llcrypt unregistered [ 6167.799439] Key type ._llcrypt unregistered [ 6184.183716] Key type ._llcrypt registered [ 6184.185493] Key type .llcrypt registered [ 6184.260269] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 6184.658585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6184.707235] alg: No test for adler32 (adler32-zlib) [ 6185.599288] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 6185.708829] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 6187.319198] Key type lgssc registered [ 6187.765890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6191.128432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6191.137301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6192.286611] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6192.300959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6192.388802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6194.063766] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6197.958878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6198.010764] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6198.026828] Lustre: 210218:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6198.169070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6199.944627] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6202.229511] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6203.742659] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6206.888976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6206.984514] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6207.068490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6207.533923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6209.395534] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6211.626807] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6213.306989] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6223.111463] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6232.912797] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6241.727889] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6250.546395] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6273.964555] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6273.971781] Lustre: Skipped 2 previous similar messages [ 6315.126787] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6315.130271] Lustre: Skipped 4 previous similar messages [ 6382.755987] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6382.759881] Lustre: Skipped 8 previous similar messages [ 6386.656233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6386.659307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6386.661995] LustreError: Skipped 1 previous similar message [ 6386.667585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6386.670715] Lustre: Skipped 1 previous similar message [ 6390.494324] Lustre: server umount lustre-OST0000 complete [ 6392.800764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6392.803410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6392.807338] Lustre: Skipped 1 previous similar message [ 6392.809444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6396.897618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6396.897655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6396.909654] Lustre: Skipped 1 previous similar message [ 6398.188417] Lustre: server umount lustre-MDT0000 complete [ 6399.785588] LustreError: 212449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475722 with bad export cookie 12530445472874640135 [ 6399.786189] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6399.792581] LustreError: 212449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6399.952309] Lustre: server umount lustre-MDT0001 complete [ 6404.609514] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 6405.862761] Key type lgssc unregistered [ 6406.013575] LNet: 214647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6406.023316] LNetError: 214647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6406.035678] LNet: Removed LNI 192.168.202.140@tcp [ 6406.447428] Key type .llcrypt unregistered [ 6406.448725] Key type ._llcrypt unregistered [ 6417.308906] Key type ._llcrypt registered [ 6417.309989] Key type .llcrypt registered [ 6417.349956] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 6417.807292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6417.847167] alg: No test for adler32 (adler32-zlib) [ 6418.738209] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 6418.835438] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 6420.431156] Key type lgssc registered [ 6421.049400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6424.381554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6424.390038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.597500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6427.224679] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6431.086952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.303331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6433.288727] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6435.635294] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6437.148115] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6440.184607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6440.347618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6442.691402] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6444.978718] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6450.599250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6471.136892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6471.139941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6471.143366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6471.147935] Lustre: Skipped 1 previous similar message [ 6471.152872] Lustre: Skipped 1 previous similar message [ 6473.284220] Lustre: server umount lustre-OST0000 complete [ 6476.256872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6476.257799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6476.262187] Lustre: Skipped 1 previous similar message [ 6481.339673] Lustre: server umount lustre-MDT0000 complete [ 6481.377447] LustreError: 216021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6481.389681] LustreError: 216021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6482.955853] LustreError: 216000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475805 with bad export cookie 12190576046612214747 [ 6482.959414] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6482.962636] LustreError: 216000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6483.136089] Lustre: server umount lustre-MDT0001 complete [ 6487.370838] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 6488.580850] Key type lgssc unregistered [ 6488.723660] LNet: 219374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6488.727200] LNetError: 219374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6488.738672] LNet: Removed LNI 192.168.202.140@tcp [ 6489.125293] Key type .llcrypt unregistered [ 6489.126318] Key type ._llcrypt unregistered [ 6499.432301] Key type ._llcrypt registered [ 6499.433741] Key type .llcrypt registered [ 6499.492265] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 6499.938404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6499.955553] alg: No test for adler32 (adler32-zlib) [ 6500.840194] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 6500.945713] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 6502.535144] Key type lgssc registered [ 6503.017864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6768.838785] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 6769.853603] Key type lgssc unregistered [ 6769.980356] LNet: 221340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6769.983309] LNetError: 221340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6769.996421] LNet: Removed LNI 192.168.202.140@tcp [ 6770.277405] Key type .llcrypt unregistered [ 6770.278762] Key type ._llcrypt unregistered [ 6778.081523] Key type ._llcrypt registered [ 6778.082523] Key type .llcrypt registered [ 6778.117392] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 6784.241281] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 6784.543640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6784.581044] alg: No test for adler32 (adler32-zlib) [ 6785.439783] Lustre: Lustre: Build Version: 2.17.50_195_gf35968a [ 6785.519199] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 6787.095150] Key type lgssc registered [ 6787.444430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6796.925661] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6502 sec ========== 13:28:38 (1772476118) [ 6797.426672] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:28:39 (1772476119) === [ 6798.467742] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:28:40 (1772476120) === [ 6811.145343] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 6812.120706] Key type lgssc unregistered [ 6812.244351] LNet: 226495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6812.247488] LNetError: 226495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6812.256308] LNet: Removed LNI 192.168.202.140@tcp [ 6812.540439] Key type .llcrypt unregistered [ 6812.541688] Key type ._llcrypt unregistered