[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 476787649 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2833800K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002410] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007643] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010096] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.013004] SELinux: Initializing. [ 0.014078] *** VALIDATE selinux *** [ 0.023550] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028732] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031115] *** VALIDATE tmpfs *** [ 0.033231] *** VALIDATE proc *** [ 0.034256] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036274] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038133] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040031] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044013] debug: unmapping init [mem 0xffffffffb9a59000-0xffffffffb9a60fff] [ 0.046946] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047663] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048022] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050013] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055317] rcu: Hierarchical SRCU implementation. [ 0.057463] smp: Bringing up secondary CPUs ... [ 0.058598] x86: Booting SMP configuration: [ 0.059030] .... node #0, CPUs: #1 #2 #3 [ 0.062369] smp: Brought up 1 node, 4 CPUs [ 0.064011] smpboot: Max logical packages: 1 [ 0.065015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141025] node 0 deferred pages initialised in 73ms [ 0.144366] devtmpfs: initialized [ 0.145225] x86/mm: Memory block size: 128MB [ 0.147718] gcov: version magic: 0x41383552 [ 0.148606] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152321] pinctrl core: initialized pinctrl subsystem [ 0.154132] [ 0.154555] ************************************************************* [ 0.156010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158011] ** ** [ 0.159009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161010] ** ** [ 0.163011] ** This means that this kernel is built to expose internal ** [ 0.164009] ** IOMMU data structures, which may compromise security on ** [ 0.166010] ** your system. ** [ 0.168011] ** ** [ 0.170010] ** If you see this message and you are not debugging the ** [ 0.171009] ** kernel, report this immediately to your vendor! ** [ 0.173012] ** ** [ 0.175011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177009] ************************************************************* [ 0.178781] NET: Registered protocol family 16 [ 0.180478] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.185060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.187437] cpuidle: using governor menu [ 0.188840] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.190516] PCI: Using configuration type 1 for base access [ 0.193128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.202047] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.204039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.208029] cryptd: max_cpu_qlen set to 1000 [ 0.210141] ACPI: Added _OSI(Module Device) [ 0.212016] ACPI: Added _OSI(Processor Device) [ 0.213010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.215013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.220169] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226201] ACPI: Interpreter enabled [ 0.227061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.229013] ACPI: Using IOAPIC for interrupt routing [ 0.231092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.234513] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.243709] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.246034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.248017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.251088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.255296] acpiphp: Slot [2] registered [ 0.256092] acpiphp: Slot [5] registered [ 0.257085] acpiphp: Slot [6] registered [ 0.258082] acpiphp: Slot [7] registered [ 0.259085] acpiphp: Slot [8] registered [ 0.260114] acpiphp: Slot [9] registered [ 0.261131] acpiphp: Slot [10] registered [ 0.262090] acpiphp: Slot [3] registered [ 0.264089] acpiphp: Slot [4] registered [ 0.265104] acpiphp: Slot [11] registered [ 0.266079] acpiphp: Slot [12] registered [ 0.267087] acpiphp: Slot [13] registered [ 0.268110] acpiphp: Slot [14] registered [ 0.269083] acpiphp: Slot [15] registered [ 0.271206] acpiphp: Slot [16] registered [ 0.272072] acpiphp: Slot [17] registered [ 0.273090] acpiphp: Slot [18] registered [ 0.274084] acpiphp: Slot [19] registered [ 0.276102] acpiphp: Slot [20] registered [ 0.277102] acpiphp: Slot [21] registered [ 0.279121] acpiphp: Slot [22] registered [ 0.280096] acpiphp: Slot [23] registered [ 0.282097] acpiphp: Slot [24] registered [ 0.283088] acpiphp: Slot [25] registered [ 0.284076] acpiphp: Slot [26] registered [ 0.285089] acpiphp: Slot [27] registered [ 0.286101] acpiphp: Slot [28] registered [ 0.288085] acpiphp: Slot [29] registered [ 0.289081] acpiphp: Slot [30] registered [ 0.290081] acpiphp: Slot [31] registered [ 0.291052] PCI host bridge to bus 0000:00 [ 0.292016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.294019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.296022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.298021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.300024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.302064] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.304181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.307009] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.310300] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.320014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.324444] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.327020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.330020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.332017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.335560] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.338085] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.341048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.343896] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.349050] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.362015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.368016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.375277] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.382017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.390013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.406014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.415096] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.423015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.428017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.446014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.456942] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.464013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.472014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.491017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.501927] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.509017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.518017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.533015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.542526] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.553015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.561016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.582017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.591771] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.597013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.602015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.615016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.624423] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.627390] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.629372] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.633413] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.635212] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.640111] iommu: Default domain type: Passthrough [ 0.641413] SCSI subsystem initialized [ 0.642131] ACPI: bus type USB registered [ 0.643106] usbcore: registered new interface driver usbfs [ 0.645065] usbcore: registered new interface driver hub [ 0.646069] usbcore: registered new device driver usb [ 0.647148] pps_core: LinuxPPS API ver. 1 registered [ 0.649009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.651053] PTP clock support registered [ 0.653044] EDAC MC: Ver: 3.0.0 [ 0.656161] PCI: Using ACPI for IRQ routing [ 0.657709] NetLabel: Initializing [ 0.657930] NetLabel: domain hash size = 128 [ 0.658000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.660072] NetLabel: unlabeled traffic allowed by default [ 0.662121] vgaarb: loaded [ 0.664268] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.665011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.669937] clocksource: Switched to clocksource kvm-clock [ 0.774123] VFS: Disk quotas dquot_6.6.0 [ 0.776524] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.779219] *** VALIDATE ramfs *** [ 0.780482] *** VALIDATE hugetlbfs *** [ 0.781775] pnp: PnP ACPI init [ 0.783846] pnp: PnP ACPI: found 6 devices [ 0.809082] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.812358] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.814693] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.817008] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.819466] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.821965] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.824807] NET: Registered protocol family 2 [ 0.827143] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.832241] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.836152] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.840995] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.843707] TCP: Hash tables configured (established 65536 bind 65536) [ 0.845944] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.848421] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.850628] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.852943] NET: Registered protocol family 1 [ 0.854959] RPC: Registered named UNIX socket transport module. [ 0.856535] RPC: Registered udp transport module. [ 0.857748] RPC: Registered tcp transport module. [ 0.858967] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.860740] NET: Registered protocol family 44 [ 0.862193] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.863817] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.865418] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.867145] PCI: CLS 0 bytes, default 64 [ 0.869694] Unpacking initramfs... [ 2.273797] debug: unmapping init [mem 0xffff930bfcc54000-0xffff930bfffbffff] [ 2.277484] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.278949] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.281714] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.784188] Initialise system trusted keyrings [ 2.786034] Key type blacklist registered [ 2.788202] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.797293] zbud: loaded [ 2.800397] *** VALIDATE nfs *** [ 2.801604] *** VALIDATE nfs4 *** [ 2.803373] pstore: using deflate compression [ 2.806931] Platform Keyring initialized [ 2.934770] NET: Registered protocol family 38 [ 2.936902] Key type asymmetric registered [ 2.938826] Asymmetric key parser 'x509' registered [ 2.941057] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.944236] io scheduler mq-deadline registered [ 2.946135] io scheduler kyber registered [ 2.947976] io scheduler bfq registered [ 2.950090] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.953156] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.955838] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.958566] ACPI: Power Button [PWRF] [ 2.963690] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.970047] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.984986] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.996152] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.015453] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.047745] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.082068] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.088497] Non-volatile memory driver v1.3 [ 3.089971] Linux agpgart interface v0.103 [ 3.122131] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.124715] vda: detected capacity change from 0 to 74698752 [ 3.140390] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.143035] vdb: detected capacity change from 0 to 1073741824 [ 3.155620] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.158269] vdc: detected capacity change from 0 to 2621440000 [ 3.174601] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.177267] vdd: detected capacity change from 0 to 2621440000 [ 3.193615] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.196185] vde: detected capacity change from 0 to 4294967296 [ 3.215031] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.217860] vdf: detected capacity change from 0 to 4294967296 [ 3.226150] libphy: Fixed MDIO Bus: probed [ 3.231692] usbcore: registered new interface driver usbserial_generic [ 3.234162] usbserial: USB Serial support registered for generic [ 3.236170] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.239895] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.241485] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.243661] mousedev: PS/2 mouse device common for all mice [ 3.246030] rtc_cmos 00:05: RTC can wake from S4 [ 3.248479] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.250293] rtc_cmos 00:05: registered as rtc0 [ 3.253252] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.257055] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.257155] intel_pstate: CPU model not supported [ 3.263248] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.267671] hid: raw HID events driver (C) Jiri Kosina [ 3.269726] usbcore: registered new interface driver usbhid [ 3.271624] usbhid: USB HID core driver [ 3.273153] drop_monitor: Initializing network drop monitor service [ 3.275347] Initializing XFRM netlink socket [ 3.277236] NET: Registered protocol family 10 [ 3.281203] Segment Routing with IPv6 [ 3.282684] NET: Registered protocol family 17 [ 3.284640] mpls_gso: MPLS GSO support [ 3.290569] RAS: Correctable Errors collector initialized. [ 3.292518] AVX version of gcm_enc/dec engaged. [ 3.293838] AES CTR mode by8 optimization enabled [ 3.372487] sched_clock: Marking stable (3372415916, 0)->(4294446575, -922030659) [ 3.375848] registered taskstats version 1 [ 3.378213] Loading compiled-in X.509 certificates [ 3.380268] zswap: loaded using pool lzo/zbud [ 3.405861] Key type big_key registered [ 3.418631] Key type encrypted registered [ 3.420351] ima: No TPM chip found, activating TPM-bypass! [ 3.422752] ima: Allocated hash algorithm: sha1 [ 3.424647] ima: No architecture policies found [ 3.426415] evm: Initialising EVM extended attributes: [ 3.428357] evm: security.selinux [ 3.429667] evm: security.ima [ 3.430852] evm: security.capability [ 3.432217] evm: HMAC attrs: 0x1 [ 3.434733] rtc_cmos 00:05: setting system clock to 2026-09-03 14:01:28 UTC (1788444088) [ 3.440983] debug: unmapping init [mem 0xffffffffbaa03000-0xffffffffbabfffff] [ 3.444134] debug: unmapping init [mem 0xffffffffb9782000-0xffffffffb9a58fff] [ 3.453085] Write protecting the kernel read-only data: 28672k [ 3.456848] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 3.459051] debug: unmapping init [mem 0xffffffffb8714000-0xffffffffb87fffff] [ 3.492490] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.501546] systemd[1]: Detected virtualization kvm. [ 3.503620] systemd[1]: Detected architecture x86-64. [ 3.505588] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.530383] systemd[1]: No hostname configured. [ 3.532588] systemd[1]: Set hostname to . [ 3.535199] random: systemd: uninitialized urandom read (16 bytes read) [ 3.538134] systemd[1]: Initializing machine ID from random generator. [ 3.575825] random: ln: uninitialized urandom read (6 bytes read) [ 3.659841] random: systemd: uninitialized urandom read (16 bytes read) [ 3.662247] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.667326] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.671378] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.318917] device-mapper: uevent: version 1.0.3 [ 4.321571] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.114852] random: fast init done [ 5.133556] virtio_net virtio0 ens2: renamed from eth0 [ 5.231633] scsi host0: ata_piix [ 5.244665] scsi host1: ata_piix [ 5.246446] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.248785] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.902177] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.937820] random: crng init done [ 9.939179] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.446641] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.589192] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.861162] SELinux: Disabled at runtime. [ 11.923699] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.932350] systemd[1]: Detected virtualization kvm. [ 11.935248] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.437924] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.441199] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.445946] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.449562] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.452408] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.460148] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.467283] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.620858] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ 12.869314] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.159078] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.166918] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.278840] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.288286] EDAC sbridge: Ver: 1.1.2 [ 15.179071] Key type dns_resolver registered [ 15.482239] NFS: Registering the id_resolver key type [ 15.484378] Key type id_resolver registered [ 15.485939] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg343-server login: [ 77.614380] libcfs: loading out-of-tree module taints kernel. [ 77.683905] Key type ._llcrypt registered [ 77.685588] Key type .llcrypt registered [ 77.889927] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 96.265687] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 98.615637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.646468] alg: No test for adler32 (adler32-zlib) [ 100.445897] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 101.256846] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 102.961117] Key type lgssc registered [ 105.702527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 124.031926] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 125.909288] hrtimer: interrupt took 7582664 ns [ 166.384251] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 178.262338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 178.297031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.504750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 179.546520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 179.627520] Lustre: lustre-MDT0000: new disk, initializing [ 179.722056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.752896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 184.644441] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.953239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.090490] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 199.148188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 199.161105] Lustre: Skipped 1 previous similar message [ 199.267957] Lustre: lustre-MDT0001: new disk, initializing [ 199.345703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.364831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 199.382794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 204.219386] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.038602] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 220.468979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.082422] Lustre: lustre-OST0000: new disk, initializing [ 221.091483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 221.105524] Lustre: 8455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.181935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.958712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 224.966721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 225.037313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 228.649971] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.441853] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.655590] Lustre: lustre-OST0001: new disk, initializing [ 244.665969] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 244.679683] Lustre: 9527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 244.781664] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.872827] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.489274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 251.512884] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 251.593383] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 263.507495] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 271.810485] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 279.600691] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing check_logdir /tmp/testlogs/ [ 285.451222] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing yml_node [ 289.968485] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 292.809071] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 295.921374] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 297.979761] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 10:06:21 EDT 2026 [ 318.963852] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 321.200413] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 337.889504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 337.900354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.917354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.919989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.921165] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.949439] Lustre: Skipped 2 previous similar messages [ 343.605907] Lustre: server umount lustre-MDT0000 complete [ 347.328487] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444432 with bad export cookie 3222256843485071261 [ 347.331382] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 347.347585] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 349.159806] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.160456] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.172886] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 349.172894] Lustre: Skipped 3 previous similar messages [ 349.217971] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 352.162424] LustreError: 6524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.163205] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 352.188287] LustreError: 6524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 352.211648] Lustre: Skipped 2 previous similar messages [ 353.907168] Lustre: server umount lustre-MDT0001 complete [ 358.393679] Lustre: server umount lustre-OST0000 complete [ 362.866794] Lustre: server umount lustre-OST0001 complete [ 370.720571] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 381.855564] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 425.664323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.919574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 425.962695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 426.149226] Lustre: lustre-MDT0000: new disk, initializing [ 426.263270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.276062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 430.355396] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 442.631642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.894445] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 443.044664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 443.060509] Lustre: Skipped 1 previous similar message [ 443.155226] Lustre: lustre-MDT0001: new disk, initializing [ 443.221156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 443.279501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 443.284401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 448.386517] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 458.775750] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 464.831681] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 471.200022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 471.420546] Lustre: lustre-OST0000: new disk, initializing [ 471.423444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 471.427721] Lustre: 17056:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 471.533098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.182816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 473.205175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 473.364631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 478.328425] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 488.571224] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 495.250235] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 495.682695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 502.563636] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 502.914047] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 509.417456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 509.419514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.435107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 509.446812] Lustre: Skipped 2 previous similar messages [ 509.463443] Lustre: Skipped 1 previous similar message [ 519.136849] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 519.257301] Lustre: server umount lustre-OST0000 complete [ 524.770960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.780125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 524.792474] Lustre: Skipped 1 previous similar message [ 524.815971] Lustre: Skipped 3 previous similar messages [ 529.402077] Lustre: server umount lustre-MDT0000 complete [ 529.893870] LustreError: 15096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.915961] LustreError: 15096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 533.599095] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444618 with bad export cookie 3222256843485072794 [ 533.605796] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.621986] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 534.171748] Lustre: server umount lustre-MDT0001 complete [ 537.697252] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 10:10:21 (1788444621) [ 544.545844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.227580] LustreError: 19323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.323488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 550.374729] LustreError: 19324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.663860] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 559.561104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.017565] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 573.123599] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 579.753968] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 587.978750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 595.090889] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 603.378345] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 614.889453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 614.899692] LustreError: Skipped 1 previous similar message [ 614.905819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.927600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 620.818724] Lustre: server umount lustre-OST0000 complete [ 631.811177] Lustre: server umount lustre-MDT0000 complete [ 635.877709] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.891676] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 636.024480] LustreError: 19303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444721 with bad export cookie 3222256843485073676 [ 636.032731] LustreError: 19303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 636.039063] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.489558] Lustre: server umount lustre-MDT0001 complete [ 648.967627] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 652.175608] Key type lgssc unregistered [ 652.531584] LNet: 22860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.551892] LNetError: 22860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 652.583931] LNet: Removed LNI 192.168.203.143@tcp [ 653.841163] Key type .llcrypt unregistered [ 653.844830] Key type ._llcrypt unregistered [ 684.054334] Key type ._llcrypt registered [ 684.057711] Key type .llcrypt registered [ 684.200775] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 685.480689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 685.565138] alg: No test for adler32 (adler32-zlib) [ 686.668500] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 686.968987] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 688.641000] Key type lgssc registered [ 689.727234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 697.603993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 697.644199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.099620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 704.424365] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 709.602422] LustreError: 24275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 709.636716] LustreError: 24275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 714.442434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 714.723833] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 714.948905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 720.158182] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 729.217174] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 735.816174] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 743.398524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 743.879180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 744.896265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 749.979866] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 758.007281] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 775.649729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 775.662173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.674328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 776.948590] Lustre: server umount lustre-OST0000 complete [ 781.289559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 781.307125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 781.336488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 784.868661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 784.870281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.956906] Lustre: server umount lustre-MDT0000 complete [ 789.986933] LustreError: 24275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 790.018981] LustreError: 24275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 791.368228] LustreError: 24255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444876 with bad export cookie 4469560937740754396 [ 791.380103] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 791.385376] LustreError: 24255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 801.875456] Lustre: server umount lustre-MDT0001 complete [ 814.937663] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 818.329230] Key type lgssc unregistered [ 818.763738] LNet: 28257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.768752] LNetError: 28257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.817290] LNet: Removed LNI 192.168.203.143@tcp [ 820.707911] Key type .llcrypt unregistered [ 820.713515] Key type ._llcrypt unregistered [ 852.241464] Key type ._llcrypt registered [ 852.244641] Key type .llcrypt registered [ 852.338912] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 854.126961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 854.295609] alg: No test for adler32 (adler32-zlib) [ 855.499204] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 855.865880] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 857.616337] Key type lgssc registered [ 859.820507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 869.793828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 869.851822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.486893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.309138] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 881.640895] LustreError: 29673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 881.666882] LustreError: 29673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 886.767767] LustreError: 29674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 886.816694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.418073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 893.814499] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 902.693561] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 909.175614] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 923.705143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 924.164141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 930.828787] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 938.540692] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 941.587386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 955.170338] Lustre: server umount lustre-OST0000 complete [ 962.018150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.022231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 962.043732] Lustre: Skipped 1 previous similar message [ 962.063630] Lustre: Skipped 1 previous similar message [ 967.137575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 970.387145] Lustre: server umount lustre-MDT0000 complete [ 972.262969] Lustre: 28983:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445041/real 1788445041] req@000000006a5d1b9c x1875320441438080/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 972.268958] LustreError: 30771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 972.288864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 972.340502] LustreError: 30771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 973.282156] Lustre: 28986:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445042/real 1788445042] req@00000000ffc8aa6d x1875320441438336/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788445058 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 974.138222] LustreError: 29654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445059 with bad export cookie 1690166825077448360 [ 974.150860] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 974.154160] LustreError: 29654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 974.498914] Lustre: server umount lustre-MDT0001 complete [ 989.898833] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 992.771813] Key type lgssc unregistered [ 993.077746] LNet: 33578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.097884] LNetError: 33578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.112605] LNet: Removed LNI 192.168.203.143@tcp [ 993.965466] Key type .llcrypt unregistered [ 993.972668] Key type ._llcrypt unregistered [ 1017.917604] Key type ._llcrypt registered [ 1017.919240] Key type .llcrypt registered [ 1017.997935] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1018.926916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.061567] alg: No test for adler32 (adler32-zlib) [ 1020.330925] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1020.540038] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1022.168186] Key type lgssc registered [ 1024.187571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1033.444606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1033.469788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1035.110228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.885781] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1045.484891] LustreError: 35008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.508703] LustreError: 35008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1049.691678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1050.006670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1053.924742] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1062.852706] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1069.760369] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1079.216953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1079.744559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.765166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1086.048014] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1094.458636] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1104.355556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.358716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1104.380241] Lustre: Skipped 1 previous similar message [ 1105.933850] Lustre: server umount lustre-OST0000 complete [ 1111.523309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1111.537245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.560883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1111.568051] Lustre: Skipped 1 previous similar message [ 1114.599233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.599531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.027982] Lustre: server umount lustre-MDT0000 complete [ 1119.723170] LustreError: 35737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.761145] LustreError: 35737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1121.143920] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445206 with bad export cookie 16838420974229649256 [ 1121.146718] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1121.155519] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1121.617786] Lustre: server umount lustre-MDT0001 complete [ 1134.618829] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1137.864300] Key type lgssc unregistered [ 1138.196741] LNet: 38545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.202283] LNetError: 38545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.220782] LNet: Removed LNI 192.168.203.143@tcp [ 1139.160961] Key type .llcrypt unregistered [ 1139.163112] Key type ._llcrypt unregistered [ 1164.310689] Key type ._llcrypt registered [ 1164.314970] Key type .llcrypt registered [ 1164.419613] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1165.348990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1165.430813] alg: No test for adler32 (adler32-zlib) [ 1166.393815] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1166.618293] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1168.288152] Key type lgssc registered [ 1169.512102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1177.225591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1177.251487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.636428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.984669] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1188.843063] LustreError: 39960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.884193] LustreError: 39960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1191.565382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1191.924904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1198.001824] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1208.604485] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1215.154552] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1222.199544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1222.526143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.551125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1228.330857] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1236.815682] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1244.130361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1244.139244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.156651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1244.713950] Lustre: lustre-OST0000: Not available for connect from 192.168.203.43@tcp (stopping) [ 1247.206411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1247.206494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1247.212734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.230721] Lustre: Skipped 1 previous similar message [ 1249.848738] LustreError: 41785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1249.984348] Lustre: server umount lustre-OST0000 complete [ 1272.803727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1272.810883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1272.833860] Lustre: Skipped 1 previous similar message [ 1276.582535] Lustre: server umount lustre-MDT0000 complete [ 1277.922792] LustreError: 39955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.956979] LustreError: 39955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1280.647312] LustreError: 39942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445365 with bad export cookie 959407231660441363 [ 1280.653038] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1280.657322] LustreError: 39942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1287.365294] Lustre: server umount lustre-MDT0001 complete [ 1298.588442] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1301.764767] Key type lgssc unregistered [ 1302.106228] LNet: 43674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1302.120853] LNetError: 43674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.206655] LNet: Removed LNI 192.168.203.143@tcp [ 1304.108785] Key type .llcrypt unregistered [ 1304.110678] Key type ._llcrypt unregistered [ 1327.478413] Key type ._llcrypt registered [ 1327.481470] Key type .llcrypt registered [ 1327.590640] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1328.901468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1329.179975] alg: No test for adler32 (adler32-zlib) [ 1330.347449] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1330.742755] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1332.424644] Key type lgssc registered [ 1334.287612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1344.012640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1344.053829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.423178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.434703] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1355.748683] LustreError: 45095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.785945] LustreError: 45095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1359.713910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1360.104240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1365.089208] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1373.256880] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1379.085550] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1387.287369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1387.795279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1390.921431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1395.287280] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1405.548459] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1412.669507] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.43@tcp (stopping) [ 1416.675926] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.685405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1416.698604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1416.719089] Lustre: Skipped 1 previous similar message [ 1417.759314] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.43@tcp (stopping) [ 1417.775540] Lustre: Skipped 2 previous similar messages [ 1418.870524] Lustre: server umount lustre-MDT0000 complete [ 1421.805685] LustreError: 45823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1421.832154] LustreError: 45823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1422.672144] LustreError: 45076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445507 with bad export cookie 11228805862073825496 [ 1422.696346] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1422.699708] LustreError: 45076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1422.864768] LustreError: 45092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.916416] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.43@tcp (stopping) [ 1426.915580] LustreError: 45090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.918893] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1426.969846] Lustre: Skipped 1 previous similar message [ 1426.982199] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1429.238311] Lustre: server umount lustre-MDT0001 complete [ 1450.502651] Lustre: server umount lustre-OST0000 complete [ 1468.469863] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1471.626288] Key type lgssc unregistered [ 1471.992227] LNet: 48962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1472.000540] LNetError: 48962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1472.013856] LNet: Removed LNI 192.168.203.143@tcp [ 1472.936060] Key type .llcrypt unregistered [ 1472.938065] Key type ._llcrypt unregistered [ 1501.241357] Key type ._llcrypt registered [ 1501.243983] Key type .llcrypt registered [ 1501.511725] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1502.643610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1503.097048] alg: No test for adler32 (adler32-zlib) [ 1504.549042] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1504.917121] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1506.717493] Key type lgssc registered [ 1508.848223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1521.023086] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1521.055536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1536.800588] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1541.920371] LustreError: 50351:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 1568.546044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.883583] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1583.578314] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1749.728378] Lustre: server umount lustre-OST0000 complete [ 1767.742819] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1770.269818] Key type lgssc unregistered [ 1770.701564] LNet: 52030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.712612] LNetError: 52030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1770.737385] LNet: Removed LNI 192.168.203.143@tcp [ 1771.662536] Key type .llcrypt unregistered [ 1771.666777] Key type ._llcrypt unregistered [ 1797.251769] Key type ._llcrypt registered [ 1797.255701] Key type .llcrypt registered [ 1797.435446] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1798.483023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1798.518928] alg: No test for adler32 (adler32-zlib) [ 1799.587914] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1799.803548] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1801.497510] Key type lgssc registered [ 1802.738448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1809.438379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1809.457079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1810.881170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1814.693728] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1820.132968] LustreError: 53444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1820.148748] LustreError: 53444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1824.285511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1825.100359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1832.256476] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1840.978650] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1848.054863] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1855.514700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1855.945783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.116534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1862.616642] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1870.476302] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1874.403684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.426514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1875.213426] Lustre: server umount lustre-OST0000 complete [ 1879.523887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1879.526359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.549121] Lustre: Skipped 2 previous similar messages [ 1879.549743] Lustre: Skipped 1 previous similar message [ 1884.642615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.650425] Lustre: Skipped 2 previous similar messages [ 1889.762414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1893.344191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1893.671058] Lustre: server umount lustre-MDT0000 complete [ 1894.881926] LustreError: 53445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.899028] LustreError: 53445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1897.310478] LustreError: 54179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445982 with bad export cookie 282252673517820033 [ 1897.322816] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1897.329746] LustreError: 54179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1897.729726] Lustre: server umount lustre-MDT0001 complete [ 1908.955730] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1912.104343] Key type lgssc unregistered [ 1912.439880] LNet: 56979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1912.445863] LNetError: 56979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1912.468681] LNet: Removed LNI 192.168.203.143@tcp [ 1913.332773] Key type .llcrypt unregistered [ 1913.334683] Key type ._llcrypt unregistered [ 1942.324681] Key type ._llcrypt registered [ 1942.327509] Key type .llcrypt registered [ 1942.494642] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1943.403055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1943.448081] alg: No test for adler32 (adler32-zlib) [ 1944.659565] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1945.077236] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1946.809126] Key type lgssc registered [ 1948.114285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1955.742589] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1955.767576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1971.360376] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1976.480403] LustreError: 58368:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 2003.104415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.746633] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2019.579564] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2026.777582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.442801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.516788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2032.575735] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2032.621675] LustreError: 58393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2032.653948] LustreError: 58393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2037.731409] LustreError: 58394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2037.753400] LustreError: 58394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2040.932106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.366045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2045.942238] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2054.104160] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2060.811844] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2063.332187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2063.351287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.379342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2066.916679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2066.917220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2066.921877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.274564] Lustre: server umount lustre-OST0000 complete [ 2097.639971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.652575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.662276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2097.666325] Lustre: Skipped 1 previous similar message [ 2097.688340] Lustre: Skipped 2 previous similar messages [ 2099.250588] Lustre: server umount lustre-MDT0000 complete [ 2102.756620] LustreError: 59406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2102.788323] LustreError: 59406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2103.978111] LustreError: 58376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446189 with bad export cookie 6226336907860335671 [ 2103.996352] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2110.533639] Lustre: server umount lustre-MDT0001 complete [ 2123.204510] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2126.714186] Key type lgssc unregistered [ 2127.063846] LNet: 62116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.072457] LNetError: 62116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2127.095647] LNet: Removed LNI 192.168.203.143@tcp [ 2128.202202] Key type .llcrypt unregistered [ 2128.204662] Key type ._llcrypt unregistered [ 2154.770419] Key type ._llcrypt registered [ 2154.772385] Key type .llcrypt registered [ 2154.880972] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2156.444895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2156.480365] alg: No test for adler32 (adler32-zlib) [ 2157.613939] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2158.002567] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2159.712382] Key type lgssc registered [ 2161.027151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2169.594866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2169.624683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.208383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2176.197239] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2181.613146] LustreError: 63529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2181.637083] LustreError: 63529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2185.410759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.715593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2190.575379] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2198.502888] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2203.978035] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2211.121863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2211.314517] LustreError: 65348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.534854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.689051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2217.971235] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2225.730608] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2247.653154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.655069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2247.660335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2247.670060] Lustre: Skipped 1 previous similar message [ 2247.693200] LustreError: Skipped 1 previous similar message [ 2248.811601] Lustre: server umount lustre-OST0000 complete [ 2252.769859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.793616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2252.796980] Lustre: Skipped 1 previous similar message [ 2257.379598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2257.387329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.408041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2258.898983] Lustre: server umount lustre-MDT0000 complete [ 2263.014615] LustreError: 63530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.034452] LustreError: 63530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2263.244742] LustreError: 63510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446348 with bad export cookie 10564347080702544399 [ 2263.251571] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2263.260741] LustreError: 63510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2263.821269] Lustre: server umount lustre-MDT0001 complete [ 2274.339583] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2276.676610] Key type lgssc unregistered [ 2276.934252] LNet: 67092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.942388] LNetError: 67092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2277.992890] LNet: Removed LNI 192.168.203.143@tcp [ 2278.691598] Key type .llcrypt unregistered [ 2278.696444] Key type ._llcrypt unregistered [ 2311.212445] Key type ._llcrypt registered [ 2311.219799] Key type .llcrypt registered [ 2311.367290] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2312.725970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2312.739694] alg: No test for adler32 (adler32-zlib) [ 2313.753735] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2313.945555] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2315.673523] Key type lgssc registered [ 2316.827581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2324.167233] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 10:40:08 (1788446408) [ 2330.200268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2330.215592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.581308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2335.971737] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2341.869432] LustreError: 68798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.905837] LustreError: 68798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2344.539679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.965167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2349.093821] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2356.450676] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2362.706260] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2369.684789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2370.029455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.087260] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2376.191329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2383.261407] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2392.900127] Lustre: Failing over lustre-MDT0000 [ 2393.194964] Lustre: server umount lustre-MDT0000 complete [ 2394.598352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.608902] LustreError: 70444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.628827] Lustre: Skipped 1 previous similar message [ 2394.657751] LustreError: 70444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2396.222973] LustreError: 70444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.715776] LustreError: 69525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.740992] LustreError: 69525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2402.207928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.352392] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2402.489814] Lustre: *** cfs_fail_loc=135, val=0*** [ 2402.566061] LustreError: 71886:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2402.571812] LustreError: 71886:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.143@tcp: cfg command failed: rc = -2 [ 2402.577326] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2402.577326] [ 2402.584974] LustreError: MGC192.168.203.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2402.595421] LustreError: 71874:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2402.601023] LustreError: 71874:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2402.606299] LustreError: 71874:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2402.611978] LustreError: 71874:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2402.626417] Lustre: server umount lustre-MDT0000 complete [ 2402.629658] LustreError: 71874:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2404.837130] LustreError: 68793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.851930] LustreError: 68793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2408.715919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.867505] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2409.054785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2409.114569] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2411.541134] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2413.537925] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2414.071880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2414.121853] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2414.196941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2424.295260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.303251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2424.311338] Lustre: Skipped 2 previous similar messages [ 2424.317110] Lustre: Skipped 1 previous similar message [ 2427.266032] Lustre: server umount lustre-OST0000 complete [ 2434.530341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.551294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2434.552889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2437.811256] Lustre: server umount lustre-MDT0000 complete [ 2439.651812] LustreError: 71288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2439.667738] LustreError: 71288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2441.110420] LustreError: 68779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446526 with bad export cookie 6804660293423055758 [ 2441.111741] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2441.124697] LustreError: 68779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2441.426817] Lustre: server umount lustre-MDT0001 complete [ 2451.662879] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2454.593944] Key type lgssc unregistered [ 2454.901344] LNet: 73667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2454.911460] LNetError: 73667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2454.936357] LNet: Removed LNI 192.168.203.143@tcp [ 2455.877355] Key type .llcrypt unregistered [ 2455.881624] Key type ._llcrypt unregistered [ 2476.853536] Key type ._llcrypt registered [ 2476.857883] Key type .llcrypt registered [ 2476.980643] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2478.136826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2478.341033] alg: No test for adler32 (adler32-zlib) [ 2479.465528] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2479.791060] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2481.464312] Key type lgssc registered [ 2482.692830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2491.749356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2491.792238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.399398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.965068] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2503.652813] LustreError: 75059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.670247] LustreError: 75059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2505.386225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2505.796129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2510.232948] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2518.122515] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2524.254379] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2531.160460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2531.641393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.244461] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2538.810795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2544.217389] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2558.323483] Lustre: Failing over lustre-MDT0000 [ 2560.606404] Lustre: server umount lustre-MDT0000 complete [ 2561.074994] LustreError: 75052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.516336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2561.535587] Lustre: Skipped 2 previous similar messages [ 2566.163765] LustreError: 75056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2566.205315] LustreError: 75056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2569.182492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.341780] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2569.469921] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2569.556507] LustreError: 78169:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2569.561781] LustreError: 78169:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.143@tcp: cfg command failed: rc = -9 [ 2569.570494] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2569.570494] [ 2569.578575] LustreError: MGC192.168.203.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2569.595245] LustreError: 78157:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2569.601336] LustreError: 78157:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2569.615274] LustreError: 78157:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2569.625660] LustreError: 78157:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2569.658023] Lustre: server umount lustre-MDT0000 complete [ 2569.660141] LustreError: 78157:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2571.295721] LustreError: 75053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2571.310601] LustreError: 75053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2575.448305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.558993] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2575.835082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2575.873798] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2576.402965] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2580.075926] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2580.981096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2581.011773] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2581.064841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2591.215247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.221429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2591.223922] Lustre: Skipped 1 previous similar message [ 2593.607973] Lustre: server umount lustre-OST0000 complete [ 2601.442382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2601.442421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.450876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2601.453520] Lustre: Skipped 1 previous similar message [ 2603.470603] Lustre: server umount lustre-MDT0000 complete [ 2606.564855] LustreError: 77550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.580581] LustreError: 77550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2608.128801] LustreError: 75792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446693 with bad export cookie 112547616414526486 [ 2608.161944] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2608.167964] LustreError: 75792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2608.630782] Lustre: server umount lustre-MDT0001 complete [ 2622.254391] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2625.067557] Key type lgssc unregistered [ 2625.406607] LNet: 79953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2625.414774] LNetError: 79953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2625.431571] LNet: Removed LNI 192.168.203.143@tcp [ 2626.488464] Key type .llcrypt unregistered [ 2626.490603] Key type ._llcrypt unregistered [ 2640.086732] Key type ._llcrypt registered [ 2640.089202] Key type .llcrypt registered [ 2640.211793] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2640.576573] Key type .llcrypt unregistered [ 2640.579919] Key type ._llcrypt unregistered [ 2660.909699] Key type ._llcrypt registered [ 2660.912309] Key type .llcrypt registered [ 2660.994966] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2661.914276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2661.946650] alg: No test for adler32 (adler32-zlib) [ 2663.002967] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2663.270355] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2664.984236] Key type lgssc registered [ 2666.006827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2673.588419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2673.600915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.944090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2679.376255] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2684.393551] LustreError: 81854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.409684] LustreError: 81854:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2687.191944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.594602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2691.480752] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2697.905646] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2702.359365] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2708.161731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2708.536653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.032320] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2719.796299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2721.161889] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2729.953116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2729.959447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.962567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2729.980519] Lustre: Skipped 1 previous similar message [ 2735.079336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2735.088982] Lustre: Skipped 1 previous similar message [ 2736.128322] Lustre: server umount lustre-OST0000 complete [ 2743.778839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2743.798647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.822358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2743.827583] Lustre: Skipped 1 previous similar message [ 2745.316724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.923826] Lustre: server umount lustre-MDT0000 complete [ 2750.437474] LustreError: 81850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.459677] LustreError: 81850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2750.471436] LustreError: 81835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446835 with bad export cookie 1067923029774110736 [ 2750.485123] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2750.487790] LustreError: 81835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2750.780601] Lustre: server umount lustre-MDT0001 complete [ 2762.696944] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2765.779898] Key type lgssc unregistered [ 2766.048806] LNet: 85390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2766.053361] LNetError: 85390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2766.069097] LNet: Removed LNI 192.168.203.143@tcp [ 2767.030667] Key type .llcrypt unregistered [ 2767.033124] Key type ._llcrypt unregistered [ 2790.562138] Key type ._llcrypt registered [ 2790.564177] Key type .llcrypt registered [ 2790.643608] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2791.789310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2791.800044] alg: No test for adler32 (adler32-zlib) [ 2792.963112] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2793.160140] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2794.776145] Key type lgssc registered [ 2795.636406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2803.636653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2803.657790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.153957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2810.378988] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2815.463770] LustreError: 86811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2815.482085] LustreError: 86811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2819.265986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.728330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2824.151294] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2831.579468] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2836.972471] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2842.983321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2843.271178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.970055] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2850.419194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2855.714680] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2864.319640] Lustre: server umount lustre-OST0000 complete [ 2870.753820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2870.769319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2870.797256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2876.387467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2876.398578] Lustre: Skipped 3 previous similar messages [ 2877.522326] Lustre: server umount lustre-MDT0000 complete [ 2881.522494] LustreError: 87540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2881.543638] LustreError: 86792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446966 with bad export cookie 5871006323231346564 [ 2881.547655] LustreError: 87540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2881.551795] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2881.570130] LustreError: 86792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2892.006295] Lustre: server umount lustre-MDT0001 complete [ 2902.801359] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2905.483314] Key type lgssc unregistered [ 2905.782341] LNet: 90349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2905.788662] LNetError: 90349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2905.824297] LNet: Removed LNI 192.168.203.143@tcp [ 2906.655561] Key type .llcrypt unregistered [ 2906.659270] Key type ._llcrypt unregistered [ 2929.218403] Key type ._llcrypt registered [ 2929.224646] Key type .llcrypt registered [ 2929.334954] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2930.580984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2930.621562] alg: No test for adler32 (adler32-zlib) [ 2931.682117] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2931.902980] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2933.552647] Key type lgssc registered [ 2934.832408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2942.000475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2942.019738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2943.427245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.515661] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2953.698709] LustreError: 91742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.716773] LustreError: 91742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2955.217201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.559620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2959.985777] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2967.579554] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2972.995179] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2979.263470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2979.610492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.489945] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2988.857363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2994.119960] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3005.926677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3005.940448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.961070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3006.434701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3006.446606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.463225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3009.511532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3014.626747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3014.635194] Lustre: Skipped 1 previous similar message [ 3019.232408] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3019.386591] Lustre: server umount lustre-OST0000 complete [ 3024.866714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3024.878617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3024.885611] Lustre: Skipped 1 previous similar message [ 3029.449960] Lustre: server umount lustre-MDT0000 complete [ 3029.987217] LustreError: 92966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.016949] LustreError: 92966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3033.210406] LustreError: 91725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447118 with bad export cookie 12313323992956825386 [ 3033.216965] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3033.222428] LustreError: 91725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3033.609521] Lustre: server umount lustre-MDT0001 complete [ 3044.779645] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 3047.181077] Key type lgssc unregistered [ 3047.413829] LNet: 95280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3047.437912] LNetError: 95280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3047.457475] LNet: Removed LNI 192.168.203.143@tcp [ 3048.240913] Key type .llcrypt unregistered [ 3048.243492] Key type ._llcrypt unregistered [ 3071.061951] Key type ._llcrypt registered [ 3071.067323] Key type .llcrypt registered [ 3071.264241] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3072.266275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3072.295656] alg: No test for adler32 (adler32-zlib) [ 3073.377939] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3073.692322] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 3075.496273] Key type lgssc registered [ 3077.136273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3085.392150] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3085.426111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3101.088922] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3106.208374] LustreError: 96666:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 3132.899628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3139.655254] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3148.166746] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3154.598329] Lustre: server umount lustre-OST0000 complete [ 3162.599720] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 10:54:06 (1788447246) [ 3169.123606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.664119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3173.752689] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3174.886353] LustreError: 98306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3174.904331] LustreError: 98306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3178.986758] LustreError: 98307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3182.414432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.860888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3187.229761] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3194.831223] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3201.154161] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3207.067699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3207.427556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.449706] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3214.660468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3220.735555] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3234.277656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3234.289858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3234.325771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3235.297833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3235.300445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3235.309661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3235.345083] Lustre: Skipped 1 previous similar message [ 3238.154857] Lustre: server umount lustre-OST0000 complete [ 3244.513759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3244.528295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.551909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3248.003340] Lustre: server umount lustre-MDT0000 complete [ 3250.657258] LustreError: 100956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.671405] LustreError: 100956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3251.652171] LustreError: 100148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447336 with bad export cookie 13611326691404006695 [ 3251.668651] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3251.681369] LustreError: 100148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3252.012368] Lustre: server umount lustre-MDT0001 complete [ 3262.226571] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 3265.632059] Key type lgssc unregistered [ 3265.911609] LNet: 102000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3265.925562] LNetError: 102000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3265.955525] LNet: Removed LNI 192.168.203.143@tcp [ 3267.246347] Key type .llcrypt unregistered [ 3267.249594] Key type ._llcrypt unregistered [ 3309.573615] Key type ._llcrypt registered [ 3309.576037] Key type .llcrypt registered [ 3309.674459] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3310.534787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3310.618648] alg: No test for adler32 (adler32-zlib) [ 3311.714691] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3312.021265] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 3313.768883] Key type lgssc registered [ 3315.085337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3323.369397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3323.397489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.918336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.014711] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3334.119278] LustreError: 104166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3334.137232] LustreError: 104166:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3338.395852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3338.782627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3342.881943] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3349.446363] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3354.247442] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3360.711870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3361.287177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.905390] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3369.471814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3378.192837] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3388.898362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3388.922418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.963414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3389.922057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3389.924177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3389.955435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3391.590653] Lustre: server umount lustre-OST0000 complete [ 3400.162923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3400.166602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3400.175303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3400.184090] Lustre: Skipped 1 previous similar message [ 3400.201419] Lustre: Skipped 2 previous similar messages [ 3401.735543] Lustre: server umount lustre-MDT0000 complete [ 3405.289023] LustreError: 105211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3405.301561] LustreError: 105211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3405.334525] LustreError: 104146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447490 with bad export cookie 16225912164361337984 [ 3405.341795] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3405.348818] LustreError: 104146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3405.645200] Lustre: server umount lustre-MDT0001 complete [ 3415.572870] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 3418.573641] Key type lgssc unregistered [ 3418.866590] LNet: 107702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3418.870986] LNetError: 107702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3418.879538] LNet: Removed LNI 192.168.203.143@tcp [ 3419.657134] Key type .llcrypt unregistered [ 3419.659110] Key type ._llcrypt unregistered [ 3440.222502] Key type ._llcrypt registered [ 3440.228668] Key type .llcrypt registered [ 3440.354774] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3441.335741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3441.367678] alg: No test for adler32 (adler32-zlib) [ 3442.762427] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3443.167541] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 3444.944172] Key type lgssc registered [ 3445.984692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3452.739492] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3452.764978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3468.384955] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3473.504363] LustreError: 108953:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 3500.128186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3505.987587] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3513.208891] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3519.189975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3519.384728] LustreError: MGC192.168.203.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3519.414696] LustreError: 109978:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3519.423774] LustreError: 109978:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3519.438539] LustreError: 109978:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3519.521202] Lustre: server umount lustre-MDT0000 complete [ 3519.523499] LustreError: 109978:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3536.459775] Lustre: server umount lustre-OST0000 complete [ 3545.712324] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 3555.328173] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3590.047199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3590.249542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3590.287930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3590.455650] Lustre: lustre-MDT0000: new disk, initializing [ 3590.582163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3590.593805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3595.594498] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3606.974633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.102770] Lustre: 112784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3607.167558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3607.171335] Lustre: Skipped 1 previous similar message [ 3607.319384] Lustre: lustre-MDT0001: new disk, initializing [ 3607.480304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3607.519324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3607.537579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3613.307101] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3622.692180] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3629.106175] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3635.850835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3636.096089] Lustre: lustre-OST0000: new disk, initializing [ 3636.099705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3636.104942] Lustre: 114753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3636.204136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3641.426879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3641.443444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3641.532932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3642.143694] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3651.106980] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3655.865870] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3656.129983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3661.484763] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3661.747941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3667.425821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3667.433873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3667.444701] Lustre: Skipped 1 previous similar message [ 3669.899123] Lustre: server umount lustre-OST0000 complete [ 3674.082585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3674.098884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.113746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3674.118709] Lustre: Skipped 1 previous similar message [ 3677.674321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3677.677617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3679.779862] Lustre: server umount lustre-MDT0000 complete [ 3683.089365] LustreError: 114752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447768 with bad export cookie 2572154503351148821 [ 3683.094847] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3683.102608] LustreError: 114752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3683.318261] Lustre: server umount lustre-MDT0001 complete [ 3691.495950] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 11:02:55 (1788447775) [ 3695.829658] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3713.570889] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 3721.668373] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3752.216046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.543346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3752.583708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3752.747906] Lustre: lustre-MDT0000: new disk, initializing [ 3752.895164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3752.911656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3757.799874] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3769.186478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3769.320981] Lustre: 119653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3769.389912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3769.397920] Lustre: Skipped 1 previous similar message [ 3769.462563] Lustre: lustre-MDT0001: new disk, initializing [ 3769.570952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3769.611214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3769.628427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3774.162635] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3784.822985] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3790.957644] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3799.925322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3800.369820] Lustre: lustre-OST0000: new disk, initializing [ 3800.373984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3800.386790] Lustre: 121622:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3800.539313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3805.855727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3805.905072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3806.175206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3808.210254] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3817.125902] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3821.506823] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3821.690771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3826.127371] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3826.286810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3831.779313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.811940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3831.820407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3831.835113] Lustre: Skipped 2 previous similar messages [ 3834.227621] Lustre: server umount lustre-OST0000 complete [ 3841.506500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3841.528199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3841.549745] Lustre: Skipped 1 previous similar message [ 3841.557204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3844.622086] Lustre: server umount lustre-MDT0000 complete [ 3847.140315] LustreError: 119663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3847.173366] LustreError: 119663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3848.412097] LustreError: 119645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447933 with bad export cookie 2572154503351149703 [ 3848.419633] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3848.421833] LustreError: 119645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3848.792828] Lustre: server umount lustre-MDT0001 complete [ 3856.411848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3856.823624] LustreError: 123838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3856.904856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3860.638644] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3861.990066] LustreError: 123839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3866.092041] LustreError: 123838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3870.445212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3875.041965] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3884.311927] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3890.761115] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3897.243593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3903.529788] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3913.689349] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3922.494399] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3930.594469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3930.605164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3930.619275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3930.628356] Lustre: Skipped 2 previous similar messages [ 3930.634343] Lustre: Skipped 2 previous similar messages [ 3934.088671] Lustre: server umount lustre-OST0000 complete [ 3944.852567] Lustre: server umount lustre-MDT0000 complete [ 3945.959543] LustreError: 123835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3945.983988] LustreError: 123835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3948.368392] LustreError: 123819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448033 with bad export cookie 2572154503351150585 [ 3948.382204] LustreError: 123819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3948.383642] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3948.592807] Lustre: server umount lustre-MDT0001 complete [ 3959.528854] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 3962.110824] Key type lgssc unregistered [ 3962.391503] LNet: 127472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3962.400968] LNetError: 127472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3963.433945] LNet: Removed LNI 192.168.203.143@tcp [ 3964.173574] Key type .llcrypt unregistered [ 3964.176591] Key type ._llcrypt unregistered [ 3978.289763] Key type ._llcrypt registered [ 3978.291879] Key type .llcrypt registered [ 3978.383503] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 3989.672768] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3990.798721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3990.823626] alg: No test for adler32 (adler32-zlib) [ 3991.906191] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3992.158166] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 3993.848156] Key type lgssc registered [ 3995.013849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4030.445469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4030.519676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4031.794576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4031.856985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4032.051749] Lustre: lustre-MDT0000: new disk, initializing [ 4032.197747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4032.224086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4037.006888] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4048.717456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4048.897615] Lustre: 130603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4049.030367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4049.039848] Lustre: Skipped 1 previous similar message [ 4049.165713] Lustre: lustre-MDT0001: new disk, initializing [ 4049.233833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4049.257950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4049.272586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4053.879660] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4066.367947] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4072.840815] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4080.023251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4080.406987] Lustre: lustre-OST0000: new disk, initializing [ 4080.413614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4080.430754] Lustre: 132575:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4080.551983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4086.750610] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4089.396158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4089.418535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4089.581696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4098.099516] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4103.627333] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4103.839601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4108.480201] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4108.697615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4115.425792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4115.426423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.432337] LustreError: Skipped 1 previous similar message [ 4115.437046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4115.438889] Lustre: Skipped 1 previous similar message [ 4115.451962] Lustre: Skipped 1 previous similar message [ 4116.756658] Lustre: server umount lustre-OST0000 complete [ 4121.064789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4121.068764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4121.079413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4125.665578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4125.666846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4130.785428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4130.788472] Lustre: Skipped 2 previous similar messages [ 4135.393197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4135.783932] Lustre: server umount lustre-MDT0000 complete [ 4139.506477] LustreError: 131400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448224 with bad export cookie 16552316488543910027 [ 4139.516360] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4139.518011] LustreError: 131400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4140.002781] Lustre: server umount lustre-MDT0001 complete [ 4148.578024] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 11:10:31 (1788448231) [ 4156.671434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4157.185498] LustreError: 134989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4157.196506] LustreError: 134989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4157.292635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4162.298283] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4162.546531] LustreError: 134990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4167.651166] LustreError: 134989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4172.416911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4172.832671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4177.433370] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4185.521944] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4191.776310] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4193.762942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4193.776367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4197.921676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4202.986747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4203.004705] Lustre: Skipped 3 previous similar messages [ 4208.096391] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4208.337321] Lustre: server umount lustre-MDT0000 complete [ 4211.691546] LustreError: 134969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448296 with bad export cookie 16552316488543910909 [ 4211.711433] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4211.721233] LustreError: 134969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4212.035809] Lustre: server umount lustre-MDT0001 complete [ 4221.364316] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 11:11:44 (1788448304) [ 4229.240838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4244.768722] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4249.888630] LustreError: 137405:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 4275.618247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4282.322480] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4291.072120] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4293.574355] Lustre: server umount lustre-OST0000 complete [ 4302.849971] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 11:13:06 (1788448386) [ 4311.267949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.825913] LustreError: 138852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4311.897118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4317.153924] LustreError: 138853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4317.542931] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4326.375315] LustreError: 138853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4326.400524] LustreError: 138853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4326.780556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4332.433315] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4340.946675] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4347.426068] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4354.659519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4355.126604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4355.133956] Lustre: Skipped 1 previous similar message [ 4362.091300] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4371.523237] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4382.701069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4382.703949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4382.724042] Lustre: Skipped 1 previous similar message [ 4382.752981] Lustre: Skipped 4 previous similar messages [ 4385.590835] Lustre: server umount lustre-MDT0000 complete [ 4387.813843] LustreError: 138852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4387.839445] LustreError: 138852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4390.022205] LustreError: 138832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448475 with bad export cookie 16552316488543911567 [ 4390.026146] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4390.032239] LustreError: 138832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4390.394771] Lustre: server umount lustre-MDT0001 complete [ 4405.576666] Lustre: server umount lustre-OST0000 complete [ 4416.215853] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 11:14:59 (1788448499) [ 4423.703335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4424.127073] LustreError: 142506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4424.197371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.474726] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4437.887814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4443.343535] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4451.946465] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4458.266458] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4465.117578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4469.494601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4471.546861] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4479.672825] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4485.218706] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4485.451585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4490.473500] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4490.688726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4494.307234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.311281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4494.332660] Lustre: Skipped 2 previous similar messages [ 4494.349915] Lustre: Skipped 1 previous similar message [ 4499.184835] Lustre: server umount lustre-OST0000 complete [ 4517.344211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4517.479974] Lustre: server umount lustre-MDT0000 complete [ 4519.905020] LustreError: 143504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.926566] LustreError: 143504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4520.862856] LustreError: 143240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448605 with bad export cookie 16552316488543912645 [ 4520.871141] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4520.872505] LustreError: 143240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4521.237814] Lustre: server umount lustre-MDT0001 complete [ 4529.225991] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 11:16:52 (1788448612) [ 4535.565052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4551.201955] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4556.320499] LustreError: 146670:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 4582.048367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4582.061387] Lustre: Skipped 2 previous similar messages [ 4588.297699] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4595.864444] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4602.925794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4603.426544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4607.803313] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4616.721723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.178734] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4629.291567] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4634.789666] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4639.022855] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4639.305640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4644.457296] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4644.708263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4647.905331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4647.913075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.917158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4647.917167] Lustre: Skipped 6 previous similar messages [ 4647.962877] Lustre: Skipped 4 previous similar messages [ 4661.216247] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4661.490263] Lustre: server umount lustre-MDT0000 complete [ 4663.277034] LustreError: 149819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4663.315599] LustreError: 149819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4665.222787] LustreError: 148448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448750 with bad export cookie 16552316488543913667 [ 4665.224994] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4665.229446] LustreError: 148448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4665.453542] Lustre: server umount lustre-MDT0001 complete [ 4679.712777] Lustre: server umount lustre-OST0000 complete [ 4688.277298] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 11:19:31 (1788448771) [ 4694.786475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4710.304843] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4715.424682] LustreError: 150868:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 4746.460758] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4753.010251] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4759.335069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4759.846945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4763.733300] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4772.336691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4776.525520] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4782.937654] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4787.657356] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4794.161470] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4794.383845] Lustre: lustre-OST0001: new disk, initializing [ 4794.388476] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4794.394908] Lustre: 153737:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4800.505440] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4801.596797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4801.605088] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4801.693935] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4809.295705] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4813.701909] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4814.004569] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4818.324444] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4818.571409] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4821.479629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4821.489628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4821.519913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4821.527379] Lustre: Skipped 8 previous similar messages [ 4826.522587] Lustre: server umount lustre-OST0000 complete [ 4836.794882] Lustre: server umount lustre-OST0001 complete [ 4846.657926] Lustre: server umount lustre-MDT0000 complete [ 4850.419475] LustreError: 153736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448935 with bad export cookie 16552316488543914619 [ 4850.421919] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4850.441266] LustreError: 153736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4850.818650] Lustre: server umount lustre-MDT0001 complete [ 4872.959869] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 11:22:36 (1788448956) [ 4874.542921] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4875.982102] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4877.745713] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 11:22:41 (1788448961) [ 4884.730414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4885.207825] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4885.261458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4885.570200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4885.583540] Lustre: Skipped 6 previous similar messages [ 4891.296084] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4900.862835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4901.035157] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4901.130252] Lustre: 157297:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4906.313198] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4914.911442] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4920.687538] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4927.877557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4928.057111] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4934.334030] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4937.843155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4942.177741] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4947.707838] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4948.004877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4952.865087] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4953.085328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4958.177454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4958.185758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4958.203157] Lustre: Skipped 5 previous similar messages [ 4961.758787] Lustre: server umount lustre-OST0000 complete [ 4985.253664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4993.786086] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4998.022143] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.042894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.047593] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4998.090608] Lustre: Skipped 1 previous similar message [ 5002.393947] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5008.238426] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5008.533292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5014.185754] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5014.527649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5021.570359] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff958f51402000.ost_server_uuid 50 [ 5023.554185] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff958f51402000.ost_server_uuid in FULL state after 0 sec [ 5023.806445] LustreError: 157305:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.43@tcp) returned error from blocking AST (req@000000003cc14a3f x1875323731054336 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000e1b03463/0xe5b5a425fd8053d5 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.43@tcp remote: 0x5e4e7507bbb6f138 expref: 5 pid: 159856 timeout: 5123 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5023.848578] LustreError: lustre-MDT0000: A client on nid 192.168.203.43@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5023.863544] LustreError: 157293:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.43@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000e1b03463/0xe5b5a425fd8053d5 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.43@tcp remote: 0x5e4e7507bbb6f138 expref: 6 pid: 159856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5029.875582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5029.887718] LustreError: Skipped 1 previous similar message [ 5035.266585] Lustre: server umount lustre-OST0000 complete [ 5046.049934] Lustre: server umount lustre-MDT0000 complete [ 5047.265311] LustreError: 157305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5047.287739] LustreError: 157305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5049.605672] LustreError: 158038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449134 with bad export cookie 16552316488543915802 [ 5049.610430] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5049.617553] LustreError: 158038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5056.109898] Lustre: server umount lustre-MDT0001 complete [ 5066.808951] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 5069.882543] Key type lgssc unregistered [ 5070.167622] LNet: 163416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5070.178070] LNetError: 163416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5070.194288] LNet: Removed LNI 192.168.203.143@tcp [ 5071.143601] Key type .llcrypt unregistered [ 5071.151732] Key type ._llcrypt unregistered [ 5097.519993] Key type ._llcrypt registered [ 5097.522416] Key type .llcrypt registered [ 5097.626579] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 5098.975369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5099.008738] alg: No test for adler32 (adler32-zlib) [ 5100.124468] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5100.400350] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 5102.112658] Key type lgssc registered [ 5103.139975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5110.771293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5110.794095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5112.222451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5116.801937] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5122.532301] LustreError: 164841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5122.567877] LustreError: 164841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5125.904205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5126.392524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5131.146516] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5139.072462] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5145.517182] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5151.671294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5152.087450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5157.263539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5158.594572] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5167.332421] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5177.480864] Lustre: Failing over lustre-MDT0000 [ 5177.740485] Lustre: server umount lustre-MDT0000 complete [ 5177.831797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5177.839613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5177.863568] LustreError: 164841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5181.922173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5181.923508] LustreError: 164842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5181.952813] Lustre: Skipped 1 previous similar message [ 5181.987713] LustreError: 164842:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5187.046550] LustreError: 164837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5187.073322] LustreError: 164837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5190.247177] LustreError: 164821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.43@tcp arrived at 1788449275 with bad export cookie 11146884401773482108 [ 5192.165683] LustreError: 164841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5192.194518] LustreError: 164841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5197.283994] Lustre: 164151:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449266/real 1788449266] req@000000004d52bbb2 x1875324892638720/t0(0) o400->MGC192.168.203.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788449282 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5197.299778] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5198.155950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5202.407120] LustreError: 164841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5202.439939] LustreError: 164841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5208.549103] LustreError: 164150:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004d52bbb2 x1875324892643712/t0(0) o250->MGC192.168.203.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5208.854860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5208.910479] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5212.751189] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5214.181385] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5214.192383] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5214.820666] Lustre: lustre-MDT0000: Denying connection for new client c0104770-5538-4de1-93f8-7d33d67135a1 (at 192.168.203.43@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5222.885317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5222.890907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5222.917731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5224.420048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5224.424729] Lustre: Skipped 1 previous similar message [ 5229.541993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5229.554470] Lustre: Skipped 1 previous similar message [ 5234.665675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5234.677815] Lustre: Skipped 1 previous similar message [ 5237.216511] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5237.496861] Lustre: server umount lustre-OST0000 complete [ 5241.354737] LustreError: 168575:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5241.372463] Lustre: 167922:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5241.384502] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5241.401688] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5241.410497] LustreError: 167922:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003d4c4df5 x1875324892659840/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5241.422210] LustreError: 167922:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5241.426863] LustreError: 167922:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5241.432374] Lustre: lustre-MDT0000: Recovery over after 0:27, of 2 clients 0 recovered and 2 were evicted. [ 5241.686996] Lustre: server umount lustre-MDT0000 complete [ 5244.902145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5244.925779] Lustre: Skipped 1 previous similar message [ 5244.933204] LustreError: 164836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5244.950993] LustreError: 164836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5245.287397] LustreError: 164822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449330 with bad export cookie 11146884401773482213 [ 5245.292697] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5245.298809] LustreError: 164822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5247.611831] Lustre: server umount lustre-MDT0001 complete [ 5257.925674] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 5260.357759] Key type lgssc unregistered [ 5260.618429] LNet: 169331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5260.629488] LNetError: 169331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5260.646054] LNet: Removed LNI 192.168.203.143@tcp [ 5261.354516] Key type .llcrypt unregistered [ 5261.357911] Key type ._llcrypt unregistered [ 5284.142780] Key type ._llcrypt registered [ 5284.149961] Key type .llcrypt registered [ 5284.251768] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 5285.215307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5285.253989] alg: No test for adler32 (adler32-zlib) [ 5286.330793] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5286.544514] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 5288.192170] Key type lgssc registered [ 5289.252898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5296.740736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5296.774423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5298.408772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5303.203758] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5308.897694] LustreError: 170744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5308.918359] LustreError: 170744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5312.220226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5312.742481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5318.420470] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5327.895860] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5335.028714] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5342.456474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5342.953272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5344.008052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5348.975536] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5356.164779] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5364.711751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5364.716349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5364.743280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5367.780795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5367.791570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5367.793257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5369.808480] Lustre: server umount lustre-OST0000 complete [ 5378.021816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.050490] Lustre: Skipped 1 previous similar message [ 5378.056683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.082224] Lustre: Skipped 2 previous similar messages [ 5383.138731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5383.159461] Lustre: Skipped 1 previous similar message [ 5387.235143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5387.239433] Lustre: Skipped 1 previous similar message [ 5389.280083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5389.454587] Lustre: server umount lustre-MDT0000 complete [ 5392.356860] LustreError: 170744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5392.376353] LustreError: 170744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5393.050650] LustreError: 170725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449478 with bad export cookie 3886564374960311381 [ 5393.055795] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5393.066218] LustreError: 170725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5393.467734] Lustre: server umount lustre-MDT0001 complete [ 5404.328627] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 5407.467501] Key type lgssc unregistered [ 5407.791971] LNet: 174280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5407.800909] LNetError: 174280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5407.829779] LNet: Removed LNI 192.168.203.143@tcp [ 5408.790414] Key type .llcrypt unregistered [ 5408.795642] Key type ._llcrypt unregistered [ 5442.071404] Key type ._llcrypt registered [ 5442.076243] Key type .llcrypt registered [ 5442.187334] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 5443.236281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5443.278061] alg: No test for adler32 (adler32-zlib) [ 5444.282775] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5444.604624] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 5446.312245] Key type lgssc registered [ 5447.418829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5454.991528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5455.018281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5456.483064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5460.736759] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5466.083546] LustreError: 175984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5466.101337] LustreError: 175984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5469.346300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5469.708710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5474.617623] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5482.781720] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5489.688000] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5496.224731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5496.650321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5501.756080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5502.157584] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5510.029626] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5524.449659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5524.450877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5524.459292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5524.474453] Lustre: Skipped 1 previous similar message [ 5524.494037] Lustre: Skipped 1 previous similar message [ 5529.634799] Lustre: server umount lustre-OST0000 complete [ 5534.693088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5534.698728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5534.704096] Lustre: Skipped 1 previous similar message [ 5534.716488] Lustre: Skipped 1 previous similar message [ 5539.785075] Lustre: server umount lustre-MDT0000 complete [ 5539.815752] LustreError: 175984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5539.831742] LustreError: 175984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5543.093853] LustreError: 175964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449628 with bad export cookie 12024293002201283420 [ 5543.101053] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5543.121223] LustreError: 175964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5543.711990] Lustre: server umount lustre-MDT0001 complete [ 5557.581724] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 5560.934717] Key type lgssc unregistered [ 5561.190028] LNet: 179546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5561.195962] LNetError: 179546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5561.203955] LNet: Removed LNI 192.168.203.143@tcp [ 5562.162303] Key type .llcrypt unregistered [ 5562.168947] Key type ._llcrypt unregistered [ 5584.685456] Key type ._llcrypt registered [ 5584.686991] Key type .llcrypt registered [ 5584.752722] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 5585.537286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5585.642337] alg: No test for adler32 (adler32-zlib) [ 5586.723462] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5586.934439] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 5588.608154] Key type lgssc registered [ 5589.755631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5602.105873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5602.127330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5603.516713] Lustre: *** cfs_fail_loc=135, val=0*** [ 5603.589935] LustreError: 181069:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5603.597275] LustreError: 181069:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.143@tcp: cfg command failed: rc = -2 [ 5603.609396] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5603.609396] [ 5603.622252] LustreError: MGC192.168.203.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5603.642175] LustreError: 181030:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5603.650739] LustreError: 181030:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5603.665918] LustreError: 181030:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5603.675911] LustreError: 181030:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5603.738262] Lustre: server umount lustre-MDT0000 complete [ 5603.742731] LustreError: 181030:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5612.114887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5612.729149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5618.113897] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5622.251311] LustreError: 181441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5622.279663] LustreError: 181441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5627.365343] LustreError: 181442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5627.785260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5628.325146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5632.866922] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5640.298710] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5645.110305] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5648.869099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5648.872526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5648.906107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5648.910067] Lustre: Skipped 1 previous similar message [ 5653.029042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5653.034663] Lustre: Skipped 1 previous similar message [ 5658.084248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5658.096944] Lustre: Skipped 1 previous similar message [ 5662.176516] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5662.331076] Lustre: server umount lustre-MDT0000 complete [ 5663.209439] LustreError: 182682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5663.234330] LustreError: 182682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5666.170985] LustreError: 181423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449751 with bad export cookie 11591564420459787883 [ 5666.185589] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5666.189552] LustreError: 181423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5666.819679] Lustre: server umount lustre-MDT0001 complete [ 5678.870173] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 5682.316465] Key type lgssc unregistered [ 5682.650688] LNet: 183768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5682.657157] LNetError: 183768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5682.678533] LNet: Removed LNI 192.168.203.143@tcp [ 5683.382277] Key type .llcrypt unregistered [ 5683.384653] Key type ._llcrypt unregistered [ 5707.303899] Key type ._llcrypt registered [ 5707.306502] Key type .llcrypt registered [ 5707.476731] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 5707.836407] Key type .llcrypt unregistered [ 5707.838112] Key type ._llcrypt unregistered [ 5725.823931] Key type ._llcrypt registered [ 5725.826730] Key type .llcrypt registered [ 5725.961324] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 5726.813942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5726.831482] alg: No test for adler32 (adler32-zlib) [ 5727.982574] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5728.214947] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 5729.928308] Key type lgssc registered [ 5731.305909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5738.572046] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5738.590647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5754.144596] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5759.265489] LustreError: 185907:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 5785.888325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.349826] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5799.753680] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5806.355284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5807.010708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5807.079047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5811.414517] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5812.199541] LustreError: 185932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5812.224264] LustreError: 185932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5817.321103] LustreError: 185933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5820.388257] LustreError: 185932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5820.431340] LustreError: 185932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5822.619557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5822.923537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5827.672598] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5836.451491] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5842.989678] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5846.204956] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5858.274344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5858.290872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.337202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5858.807485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5858.815933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.842424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5861.521496] Lustre: server umount lustre-OST0000 complete [ 5869.026277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5869.036284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5869.040942] Lustre: Skipped 1 previous similar message [ 5869.044566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5869.062427] Lustre: Skipped 1 previous similar message [ 5874.147739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5874.153183] Lustre: Skipped 1 previous similar message [ 5878.241478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5878.254639] Lustre: Skipped 1 previous similar message [ 5880.289538] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5880.487686] Lustre: server umount lustre-MDT0000 complete [ 5883.360929] LustreError: 186948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.378658] LustreError: 186948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5884.576681] LustreError: 185913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449969 with bad export cookie 17154548428031401527 [ 5884.580708] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5884.589571] LustreError: 185913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5884.930628] Lustre: server umount lustre-MDT0001 complete [ 5896.339826] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 5899.672829] Key type lgssc unregistered [ 5900.041616] LNet: 189827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5900.048681] LNetError: 189827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5900.069444] LNet: Removed LNI 192.168.203.143@tcp [ 5901.019149] Key type .llcrypt unregistered [ 5901.027507] Key type ._llcrypt unregistered [ 5928.718578] Key type ._llcrypt registered [ 5928.720832] Key type .llcrypt registered [ 5928.809051] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 5929.750200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5929.765709] alg: No test for adler32 (adler32-zlib) [ 5930.926200] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5931.212332] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 5932.904175] Key type lgssc registered [ 5934.341699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5942.451181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5942.474366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5943.766766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5948.305818] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5952.994839] LustreError: 191240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5953.026432] LustreError: 191240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5957.226764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5957.720975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5962.710839] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5970.964525] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5976.622506] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5983.118349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5983.554733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5989.688749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5990.242834] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 5999.586690] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6005.784693] Lustre: Failing over lustre-MDT0000 [ 6006.278811] Lustre: server umount lustre-MDT0000 complete [ 6007.265486] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6007.274929] LustreError: 191240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6007.284165] Lustre: Skipped 2 previous similar messages [ 6007.335468] LustreError: 191240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6012.390490] LustreError: 193599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6012.415997] LustreError: 193599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6017.507671] LustreError: 191239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6017.527497] LustreError: 191239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6022.627470] LustreError: 191240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6022.671652] LustreError: 191240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6023.649749] Lustre: 190551:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450092/real 1788450092] req@000000008fb8997a x1875325764003840/t0(0) o400->MGC192.168.203.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788450108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6023.709219] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6026.476860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6033.891861] LustreError: 193599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6033.896969] LustreError: 190548:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000041b6477d x1875325764010112/t0(0) o250->MGC192.168.203.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6033.921432] LustreError: 193599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 6034.221529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6034.268131] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6034.453955] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6038.441347] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6039.536166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6039.551766] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6039.601348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6043.011423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6044.923846] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6055.404049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6055.427221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6055.456072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6058.465523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6058.475429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6058.506979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6060.009295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6061.292893] Lustre: server umount lustre-OST0000 complete [ 6070.241854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.244773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6070.277643] Lustre: Skipped 1 previous similar message [ 6070.279457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6070.306804] Lustre: Skipped 2 previous similar messages [ 6072.004964] Lustre: server umount lustre-MDT0000 complete [ 6075.370594] LustreError: 191239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6075.396213] LustreError: 191239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6075.815646] LustreError: 191220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450160 with bad export cookie 7024221870316387696 [ 6075.827809] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6075.838497] LustreError: 191220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6076.345732] Lustre: server umount lustre-MDT0001 complete [ 6089.467485] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 6092.610858] Key type lgssc unregistered [ 6092.863637] LNet: 195992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6092.877539] LNetError: 195992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6092.893945] LNet: Removed LNI 192.168.203.143@tcp [ 6093.880591] Key type .llcrypt unregistered [ 6093.884775] Key type ._llcrypt unregistered [ 6119.755686] Key type ._llcrypt registered [ 6119.758589] Key type .llcrypt registered [ 6119.847937] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 6121.182498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6121.386297] alg: No test for adler32 (adler32-zlib) [ 6122.543852] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6122.888482] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 6124.600599] Key type lgssc registered [ 6125.667795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6133.459892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6133.473437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6134.819457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6139.374803] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6143.971688] LustreError: 197406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.981530] LustreError: 197406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6148.430839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6148.759089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6152.826345] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6161.227700] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6167.634991] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6175.127637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6175.515148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6179.653763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6182.239406] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6190.774918] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6193.612139] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6203.551667] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6219.420503] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6224.868774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6224.871935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6224.886075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6224.906302] Lustre: Skipped 1 previous similar message [ 6224.915726] Lustre: Skipped 1 previous similar message [ 6229.990908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6230.002424] Lustre: Skipped 1 previous similar message [ 6230.821945] Lustre: server umount lustre-OST0000 complete [ 6235.106548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6235.110889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6235.120819] Lustre: Skipped 1 previous similar message [ 6235.136610] Lustre: Skipped 1 previous similar message [ 6240.228336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6240.240024] Lustre: Skipped 1 previous similar message [ 6240.492623] Lustre: server umount lustre-MDT0000 complete [ 6243.693810] LustreError: 197386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450328 with bad export cookie 86244395628607995 [ 6243.701243] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6243.704340] LustreError: 197386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6244.005611] Lustre: server umount lustre-MDT0001 complete [ 6254.408900] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 6257.181204] Key type lgssc unregistered [ 6257.480683] LNet: 201272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6257.486619] LNetError: 201272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6257.506946] LNet: Removed LNI 192.168.203.143@tcp [ 6258.421453] Key type .llcrypt unregistered [ 6258.425933] Key type ._llcrypt unregistered [ 6274.421771] Key type ._llcrypt registered [ 6274.424111] Key type .llcrypt registered [ 6274.519266] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 6275.467828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6275.478743] alg: No test for adler32 (adler32-zlib) [ 6276.535545] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6276.760993] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 6278.408219] Key type lgssc registered [ 6279.510372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6286.723265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6286.737979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6288.229975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6292.341889] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6298.593950] LustreError: 202516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6298.614369] LustreError: 202516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6300.618923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6301.004864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6305.295803] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6312.459371] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6318.197555] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6323.624790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6323.940767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6329.715864] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6333.175719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6337.331996] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6344.572625] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6355.718288] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6361.512687] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6370.571342] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6381.541589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6381.557974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6381.574522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6384.611144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6384.619856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6384.622072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6384.649761] Lustre: Skipped 1 previous similar message [ 6387.603396] Lustre: server umount lustre-OST0000 complete [ 6393.317487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6393.325116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6393.346789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6397.394912] Lustre: server umount lustre-MDT0000 complete [ 6399.970617] LustreError: 202516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6399.989423] LustreError: 202516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6401.025250] LustreError: 202496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450486 with bad export cookie 7306491432711065189 [ 6401.025689] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6401.049255] LustreError: 202496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6401.342627] Lustre: server umount lustre-MDT0001 complete [ 6411.295253] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 6414.168753] Key type lgssc unregistered [ 6414.521189] LNet: 207492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6414.530858] LNetError: 207492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6414.549883] LNet: Removed LNI 192.168.203.143@tcp [ 6415.424475] Key type .llcrypt unregistered [ 6415.430918] Key type ._llcrypt unregistered [ 6439.112996] Key type ._llcrypt registered [ 6439.116669] Key type .llcrypt registered [ 6439.247099] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 6440.042629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6440.080422] alg: No test for adler32 (adler32-zlib) [ 6441.176765] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6441.435464] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 6443.128177] Key type lgssc registered [ 6444.201654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6450.798557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6450.814435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6452.302016] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6456.549236] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6462.437418] LustreError: 208904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6462.459988] LustreError: 208904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6463.954902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6464.303390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6468.536763] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6475.388973] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6480.947971] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6487.618593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6488.132627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6494.987540] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6497.332866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6502.680322] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6517.729316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6517.741858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6517.746674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6517.776516] Lustre: Skipped 1 previous similar message [ 6522.072360] Lustre: server umount lustre-OST0000 complete [ 6527.976522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6527.977966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6527.990229] Lustre: Skipped 1 previous similar message [ 6527.994904] Lustre: Skipped 1 previous similar message [ 6532.492326] Lustre: server umount lustre-MDT0000 complete [ 6533.099423] LustreError: 208904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6533.123920] LustreError: 208904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6536.760693] LustreError: 208884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450621 with bad export cookie 13492175148876305162 [ 6536.774434] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6536.779038] LustreError: 208884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6537.330382] Lustre: server umount lustre-MDT0001 complete [ 6550.082847] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 6552.841370] Key type lgssc unregistered [ 6553.132229] LNet: 212593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6553.143244] LNetError: 212593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6553.163318] LNet: Removed LNI 192.168.203.143@tcp [ 6554.051083] Key type .llcrypt unregistered [ 6554.055157] Key type ._llcrypt unregistered [ 6573.882899] Key type ._llcrypt registered [ 6573.885892] Key type .llcrypt registered [ 6573.999986] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 6575.163790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6575.230559] alg: No test for adler32 (adler32-zlib) [ 6576.365802] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6576.635142] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 6578.320167] Key type lgssc registered [ 6579.659711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6588.114500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6588.144373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6589.585107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6594.742691] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6599.140582] LustreError: 213859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6599.164443] LustreError: 213859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6603.336258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6603.666590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6608.525493] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6616.957904] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6623.539627] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6630.558888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6631.183393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6634.228342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6639.285569] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6648.915360] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6669.280946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6669.295775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.302241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6669.318096] Lustre: Skipped 1 previous similar message [ 6669.332410] Lustre: Skipped 1 previous similar message [ 6672.177513] Lustre: server umount lustre-OST0000 complete [ 6679.524912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6679.531800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6679.560440] Lustre: Skipped 1 previous similar message [ 6682.676734] Lustre: server umount lustre-MDT0000 complete [ 6684.646583] LustreError: 213860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6684.678712] LustreError: 213860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6686.798629] LustreError: 216503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450771 with bad export cookie 242493828017515879 [ 6686.817691] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6686.820655] LustreError: 216503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6687.320328] Lustre: server umount lustre-MDT0001 complete [ 6699.412160] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 6701.955327] Key type lgssc unregistered [ 6702.224442] LNet: 217548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6702.233836] LNetError: 217548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6702.261106] LNet: Removed LNI 192.168.203.143@tcp [ 6703.172178] Key type .llcrypt unregistered [ 6703.176861] Key type ._llcrypt unregistered [ 6720.212847] Key type ._llcrypt registered [ 6720.217974] Key type .llcrypt registered [ 6720.320795] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 6721.242968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6721.389258] alg: No test for adler32 (adler32-zlib) [ 6722.435906] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6722.702290] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 6724.378211] Key type lgssc registered [ 6725.320653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6731.781887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6731.829701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6733.206024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6736.978203] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6743.525083] LustreError: 218806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6743.541734] LustreError: 218806:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6744.681855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6744.943545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6749.333961] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6755.644604] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6760.414458] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6766.289681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6766.602093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6772.424570] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6778.942716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6779.313490] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6788.680503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6788.740861] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6788.908768] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6795.119371] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6801.880166] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6803.174989] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6814.640878] Lustre: Permanently deactivating lustre-OST0001 [ 6814.661768] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6824.558126] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6831.004674] Lustre: Permanently reactivating lustre-OST0001 [ 6831.026749] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6831.032166] Lustre: Skipped 2 previous similar messages [ 6832.635667] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6832.657941] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6832.666995] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6832.688213] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6832.718809] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6842.360508] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.363564] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6842.382227] Lustre: Skipped 2 previous similar messages [ 6842.419080] Lustre: Skipped 1 previous similar message [ 6847.463832] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6847.477410] Lustre: Skipped 1 previous similar message [ 6847.614019] Lustre: server umount lustre-OST0001 complete [ 6852.586773] LustreError: 220629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6852.591436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6852.594722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6852.612967] LustreError: 220629:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6852.650643] Lustre: Skipped 1 previous similar message [ 6857.748505] LustreError: 220630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6857.769502] LustreError: 220630:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6857.858725] Lustre: server umount lustre-OST0000 complete [ 6862.820643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6862.824495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6862.836745] Lustre: Skipped 1 previous similar message [ 6862.840289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6862.841426] Lustre: Skipped 2 previous similar messages [ 6867.508942] Lustre: server umount lustre-MDT0000 complete [ 6867.939685] LustreError: 219535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6867.959673] LustreError: 219535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6871.171237] LustreError: 220653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450956 with bad export cookie 1653673623040438705 [ 6871.177587] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6871.188533] LustreError: 220653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6871.591582] Lustre: server umount lustre-MDT0001 complete [ 6881.048524] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 6883.583187] Key type lgssc unregistered [ 6883.837444] LNet: 223881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6883.850895] LNetError: 223881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6884.906764] LNet: Removed LNI 192.168.203.143@tcp [ 6885.721321] Key type .llcrypt unregistered [ 6885.723831] Key type ._llcrypt unregistered [ 6921.481038] Key type ._llcrypt registered [ 6921.485798] Key type .llcrypt registered [ 6921.588680] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 6922.669601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6922.750710] alg: No test for adler32 (adler32-zlib) [ 6923.833898] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6924.107227] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 6925.816277] Key type lgssc registered [ 6927.352096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6936.543390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6936.571099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6938.022962] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6938.053965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6938.269775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6944.428652] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6953.116250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6953.256645] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6953.285893] Lustre: 226087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6953.566142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6958.193531] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6965.558296] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6970.495912] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6977.980160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6978.221186] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6978.421563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6984.639892] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6985.884388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6992.678615] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6995.750296] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7004.472954] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7012.016621] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7021.581828] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7035.672960] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7035.681675] Lustre: Skipped 1 previous similar message [ 7055.475667] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7055.486124] Lustre: Skipped 1 previous similar message [ 7094.162153] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7094.171151] Lustre: Skipped 4 previous similar messages [ 7158.189522] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7158.198466] Lustre: Skipped 7 previous similar messages [ 7175.142565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7175.154798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7175.194096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7175.650431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7175.664065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7180.408412] Lustre: server umount lustre-OST0000 complete [ 7185.900545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7185.902703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7185.921528] Lustre: Skipped 1 previous similar message [ 7185.927955] Lustre: Skipped 1 previous similar message [ 7190.272139] Lustre: server umount lustre-MDT0000 complete [ 7193.672724] LustreError: 226828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451278 with bad export cookie 1467541358073963523 [ 7193.688110] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7193.988615] Lustre: server umount lustre-MDT0001 complete [ 7203.786696] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 7206.164948] Key type lgssc unregistered [ 7206.454219] LNet: 230909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7206.466082] LNetError: 230909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7206.484489] LNet: Removed LNI 192.168.203.143@tcp [ 7207.276819] Key type .llcrypt unregistered [ 7207.278782] Key type ._llcrypt unregistered [ 7229.964641] Key type ._llcrypt registered [ 7229.967944] Key type .llcrypt registered [ 7230.068732] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 7231.047249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7231.251980] alg: No test for adler32 (adler32-zlib) [ 7232.426239] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7232.706654] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 7234.440194] Key type lgssc registered [ 7235.513758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7243.907798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7243.934300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7245.430844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7249.994691] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7255.020130] LustreError: 232325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7255.036077] LustreError: 232325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7259.052889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7259.445377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7263.778466] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7271.137572] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7276.333468] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7283.056306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7283.450474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7290.091937] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7290.613318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7298.507835] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7319.274525] Lustre: server umount lustre-OST0000 complete [ 7326.179815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7326.184746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7326.199506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7326.691888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7326.711756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7331.823914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7331.836675] Lustre: Skipped 1 previous similar message [ 7333.928583] Lustre: server umount lustre-MDT0000 complete [ 7336.416634] Lustre: 231634:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451405/real 1788451405] req@00000000549f6863 x1875327128212352/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788451421 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7336.464333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7336.490907] LustreError: 232320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7336.504227] LustreError: 232320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7337.376275] Lustre: 231637:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451406/real 1788451406] req@0000000013865c62 x1875327128213120/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788451422 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7337.874704] LustreError: 232307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451422 with bad export cookie 605631719808019925 [ 7337.889069] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7337.895443] LustreError: 232307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7338.299376] Lustre: server umount lustre-MDT0001 complete [ 7349.337925] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 7353.527857] Key type lgssc unregistered [ 7353.929166] LNet: 236020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7353.943187] LNetError: 236020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7353.966715] LNet: Removed LNI 192.168.203.143@tcp [ 7355.376379] Key type .llcrypt unregistered [ 7355.384146] Key type ._llcrypt unregistered [ 7380.144663] Key type ._llcrypt registered [ 7380.149973] Key type .llcrypt registered [ 7380.238760] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 7381.366774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7381.384257] alg: No test for adler32 (adler32-zlib) [ 7382.647073] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7383.028875] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 7384.736148] Key type lgssc registered [ 7385.959297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7657.833562] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 7660.745480] Key type lgssc unregistered [ 7661.073332] LNet: 238048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7661.083184] LNetError: 238048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7661.100556] LNet: Removed LNI 192.168.203.143@tcp [ 7662.122610] Key type .llcrypt unregistered [ 7662.124945] Key type ._llcrypt unregistered [ 7688.780591] Key type ._llcrypt registered [ 7688.782575] Key type .llcrypt registered [ 7688.873511] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 7704.456383] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 7705.560990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7705.670930] alg: No test for adler32 (adler32-zlib) [ 7706.859468] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7707.157318] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 7708.944279] Key type lgssc registered [ 7710.170750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7742.895478] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7443 sec ========== 12:10:26 (1788451826) [ 7744.745053] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:10:28 (1788451828) === [ 7748.058404] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:10:31 (1788451831) === [ 7790.561463] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 7793.499684] Key type lgssc unregistered [ 7793.828850] LNet: 243289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7793.842405] LNetError: 243289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7793.866950] LNet: Removed LNI 192.168.203.143@tcp [ 7794.775511] Key type .llcrypt unregistered [ 7794.779911] Key type ._llcrypt unregistered