[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 523376083 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003137] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.007708] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010117] LSM: Security Framework initializing [ 0.011036] Yama: becoming mindful. [ 0.012025] SELinux: Initializing. [ 0.014051] *** VALIDATE selinux *** [ 0.021709] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026146] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027136] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028097] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029095] *** VALIDATE tmpfs *** [ 0.031147] *** VALIDATE proc *** [ 0.032196] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.035238] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036138] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038025] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.042081] debug: unmapping init [mem 0xffffffff97c59000-0xffffffff97c60fff] [ 0.044265] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045623] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053287] rcu: Hierarchical SRCU implementation. [ 0.055581] smp: Bringing up secondary CPUs ... [ 0.056461] x86: Booting SMP configuration: [ 0.057015] .... node #0, CPUs: #1 #2 #3 [ 0.064501] smp: Brought up 1 node, 4 CPUs [ 0.066013] smpboot: Max logical packages: 1 [ 0.067009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.096063] node 0 deferred pages initialised in 27ms [ 0.099391] devtmpfs: initialized [ 0.100269] x86/mm: Memory block size: 128MB [ 0.103275] gcov: version magic: 0x41383552 [ 0.105130] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.106066] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.107224] pinctrl core: initialized pinctrl subsystem [ 0.108184] [ 0.108610] ************************************************************* [ 0.109015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.110016] ** ** [ 0.111011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.112013] ** ** [ 0.113013] ** This means that this kernel is built to expose internal ** [ 0.114011] ** IOMMU data structures, which may compromise security on ** [ 0.115012] ** your system. ** [ 0.116015] ** ** [ 0.117016] ** If you see this message and you are not debugging the ** [ 0.118012] ** kernel, report this immediately to your vendor! ** [ 0.119011] ** ** [ 0.120014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.121009] ************************************************************* [ 0.123687] NET: Registered protocol family 16 [ 0.124747] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.125232] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.126071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.127937] cpuidle: using governor menu [ 0.130216] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.134663] PCI: Using configuration type 1 for base access [ 0.137289] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.150046] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.151017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.154058] cryptd: max_cpu_qlen set to 1000 [ 0.156235] ACPI: Added _OSI(Module Device) [ 0.157000] ACPI: Added _OSI(Processor Device) [ 0.159019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.161014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.167000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.175747] ACPI: Interpreter enabled [ 0.177062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.179011] ACPI: Using IOAPIC for interrupt routing [ 0.182188] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.186413] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.199473] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.202051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.206024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.210070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.217056] acpiphp: Slot [2] registered [ 0.219227] acpiphp: Slot [5] registered [ 0.221258] acpiphp: Slot [6] registered [ 0.223180] acpiphp: Slot [7] registered [ 0.225123] acpiphp: Slot [8] registered [ 0.228142] acpiphp: Slot [9] registered [ 0.230153] acpiphp: Slot [10] registered [ 0.232121] acpiphp: Slot [3] registered [ 0.234696] acpiphp: Slot [4] registered [ 0.238132] acpiphp: Slot [11] registered [ 0.239149] acpiphp: Slot [12] registered [ 0.242150] acpiphp: Slot [13] registered [ 0.245139] acpiphp: Slot [14] registered [ 0.247713] acpiphp: Slot [15] registered [ 0.249344] acpiphp: Slot [16] registered [ 0.251092] acpiphp: Slot [17] registered [ 0.253103] acpiphp: Slot [18] registered [ 0.255313] acpiphp: Slot [19] registered [ 0.257101] acpiphp: Slot [20] registered [ 0.259094] acpiphp: Slot [21] registered [ 0.261211] acpiphp: Slot [22] registered [ 0.263106] acpiphp: Slot [23] registered [ 0.266114] acpiphp: Slot [24] registered [ 0.268115] acpiphp: Slot [25] registered [ 0.270117] acpiphp: Slot [26] registered [ 0.272105] acpiphp: Slot [27] registered [ 0.273127] acpiphp: Slot [28] registered [ 0.276129] acpiphp: Slot [29] registered [ 0.278109] acpiphp: Slot [30] registered [ 0.280102] acpiphp: Slot [31] registered [ 0.282127] PCI host bridge to bus 0000:00 [ 0.284060] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.287048] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.289145] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.293053] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.296027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.300027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.302239] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.306434] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.310618] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.321013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.326054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.328017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.331016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.334022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.337431] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.340960] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.344177] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.348996] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.356019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.371016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.377013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.383760] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.393015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.416016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.426185] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.434015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.441015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.458017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.475024] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.481014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.487014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.502022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.509321] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.516022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.521015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.535015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.546000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.556018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.563016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.578013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.604021] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.615021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.625016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.647015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.667889] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.671792] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.676367] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.682765] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.686157] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.690086] iommu: Default domain type: Passthrough [ 0.693738] SCSI subsystem initialized [ 0.696132] ACPI: bus type USB registered [ 0.698106] usbcore: registered new interface driver usbfs [ 0.702104] usbcore: registered new interface driver hub [ 0.707085] usbcore: registered new device driver usb [ 0.709090] pps_core: LinuxPPS API ver. 1 registered [ 0.710007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.713085] PTP clock support registered [ 0.717030] EDAC MC: Ver: 3.0.0 [ 0.719133] PCI: Using ACPI for IRQ routing [ 0.721013] NetLabel: Initializing [ 0.723103] NetLabel: domain hash size = 128 [ 0.726011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.730073] NetLabel: unlabeled traffic allowed by default [ 0.733394] vgaarb: loaded [ 0.736006] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.739014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.748000] clocksource: Switched to clocksource kvm-clock [ 0.951624] VFS: Disk quotas dquot_6.6.0 [ 0.953310] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.956310] *** VALIDATE ramfs *** [ 0.957808] *** VALIDATE hugetlbfs *** [ 0.959780] pnp: PnP ACPI init [ 0.962491] pnp: PnP ACPI: found 6 devices [ 0.979787] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.983745] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.986487] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.989324] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.992586] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.995448] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.998792] NET: Registered protocol family 2 [ 1.001715] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.007826] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.012449] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.019483] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.024329] TCP: Hash tables configured (established 65536 bind 65536) [ 1.031183] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.035793] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.039601] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.043538] NET: Registered protocol family 1 [ 1.046651] RPC: Registered named UNIX socket transport module. [ 1.049913] RPC: Registered udp transport module. [ 1.052785] RPC: Registered tcp transport module. [ 1.055211] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.058228] NET: Registered protocol family 44 [ 1.059870] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.062593] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.065516] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.068676] PCI: CLS 0 bytes, default 64 [ 1.070887] Unpacking initramfs... [ 3.292443] debug: unmapping init [mem 0xffff94853cc54000-0xffff94853ffbffff] [ 3.301677] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.309473] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.317623] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.616327] Initialise system trusted keyrings [ 4.619105] Key type blacklist registered [ 4.620914] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.630846] zbud: loaded [ 4.633622] *** VALIDATE nfs *** [ 4.634622] *** VALIDATE nfs4 *** [ 4.636282] pstore: using deflate compression [ 4.640262] Platform Keyring initialized [ 4.788557] NET: Registered protocol family 38 [ 4.790807] Key type asymmetric registered [ 4.792833] Asymmetric key parser 'x509' registered [ 4.795255] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.799169] io scheduler mq-deadline registered [ 4.800667] io scheduler kyber registered [ 4.802750] io scheduler bfq registered [ 4.804815] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.808685] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.812522] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.817476] ACPI: Power Button [PWRF] [ 4.823554] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.831458] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.855434] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.863199] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.880548] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.909113] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.939267] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.954574] Non-volatile memory driver v1.3 [ 4.956409] Linux agpgart interface v0.103 [ 4.988248] virtio_blk virtio1: [vda] 134920 512-byte logical blocks (69.1 MB/65.9 MiB) [ 4.990712] vda: detected capacity change from 0 to 69079040 [ 5.010652] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.014131] vdb: detected capacity change from 0 to 1073741824 [ 5.039649] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.051500] vdc: detected capacity change from 0 to 2621440000 [ 5.105735] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.119683] vdd: detected capacity change from 0 to 2621440000 [ 5.142383] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.146314] vde: detected capacity change from 0 to 4294967296 [ 5.166537] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.170747] vdf: detected capacity change from 0 to 4294967296 [ 5.178191] libphy: Fixed MDIO Bus: probed [ 5.184724] usbcore: registered new interface driver usbserial_generic [ 5.187510] usbserial: USB Serial support registered for generic [ 5.190130] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.195621] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.198080] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.200535] mousedev: PS/2 mouse device common for all mice [ 5.203263] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.205500] rtc_cmos 00:05: RTC can wake from S4 [ 5.220032] rtc_cmos 00:05: registered as rtc0 [ 5.222176] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.226269] intel_pstate: CPU model not supported [ 5.231476] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.236568] hid: raw HID events driver (C) Jiri Kosina [ 5.245761] usbcore: registered new interface driver usbhid [ 5.247245] usbhid: USB HID core driver [ 5.252667] drop_monitor: Initializing network drop monitor service [ 5.254137] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.255407] Initializing XFRM netlink socket [ 5.255762] NET: Registered protocol family 10 [ 5.258509] Segment Routing with IPv6 [ 5.268158] NET: Registered protocol family 17 [ 5.270060] mpls_gso: MPLS GSO support [ 5.275560] RAS: Correctable Errors collector initialized. [ 5.277706] AVX version of gcm_enc/dec engaged. [ 5.279500] AES CTR mode by8 optimization enabled [ 5.376850] sched_clock: Marking stable (5376780159, 0)->(6438157299, -1061377140) [ 5.380839] registered taskstats version 1 [ 5.383573] Loading compiled-in X.509 certificates [ 5.386310] zswap: loaded using pool lzo/zbud [ 5.413425] Key type big_key registered [ 5.431637] Key type encrypted registered [ 5.433743] ima: No TPM chip found, activating TPM-bypass! [ 5.436495] ima: Allocated hash algorithm: sha1 [ 5.438700] ima: No architecture policies found [ 5.440788] evm: Initialising EVM extended attributes: [ 5.443308] evm: security.selinux [ 5.444746] evm: security.ima [ 5.446132] evm: security.capability [ 5.447779] evm: HMAC attrs: 0x1 [ 5.450810] rtc_cmos 00:05: setting system clock to 2026-05-13 06:01:56 UTC (1778652116) [ 5.460337] debug: unmapping init [mem 0xffffffff98c03000-0xffffffff98dfffff] [ 5.466556] debug: unmapping init [mem 0xffffffff97982000-0xffffffff97c58fff] [ 5.479236] Write protecting the kernel read-only data: 28672k [ 5.483484] debug: unmapping init [mem 0xffffffff96003000-0xffffffff961fffff] [ 5.487393] debug: unmapping init [mem 0xffffffff96914000-0xffffffff969fffff] [ 5.523341] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.530658] systemd[1]: Detected virtualization kvm. [ 5.532418] systemd[1]: Detected architecture x86-64. [ 5.534302] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.564844] systemd[1]: No hostname configured. [ 5.566318] systemd[1]: Set hostname to . [ 5.568436] random: systemd: uninitialized urandom read (16 bytes read) [ 5.570694] systemd[1]: Initializing machine ID from random generator. [ 5.832737] random: ln: uninitialized urandom read (6 bytes read) [ 5.956128] random: systemd: uninitialized urandom read (16 bytes read) [ 5.958779] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.965675] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.974079] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.818176] device-mapper: uevent: version 1.0.3 [ 7.821389] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices...[ 8.816421] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ 10.015418] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.007638] scsi host0: ata_piix [ 11.296992] scsi host1: ata_piix [ 11.301767] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.307916] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.063803] random: crng init done [ 16.072938] random: 7 urandom warning(s) missed due to ratelimiting [ 19.428549] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.312989] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.463333] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.149921] SELinux: Disabled at runtime. [ 23.959722] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.971849] systemd[1]: Detected virtualization kvm. [ 23.973709] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.013221] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.016787] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.022364] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.027422] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.029882] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.042662] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.049441] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-f[ 27.115251] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS s-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 28.720720] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.777922] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.034934] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.297893] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.415447] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 35.396635] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 36.279593] NFS: Registering the id_resolver key type [ 36.285766] Key type id_resolver registered [ 36.293625] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg123-server login: [ 66.026271] hrtimer: interrupt took 5294469 ns [ 117.674531] libcfs: loading out-of-tree module taints kernel. [ 117.726475] Key type ._llcrypt registered [ 117.734464] Key type .llcrypt registered [ 117.845527] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 134.031065] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 135.376624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 135.386231] alg: No test for adler32 (adler32-zlib) [ 136.698855] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 137.490569] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 139.176205] Key type lgssc registered [ 140.530286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 153.785646] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 188.572547] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 199.887743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 199.925143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.317606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 201.368726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 201.545502] Lustre: lustre-MDT0000: new disk, initializing [ 201.676793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.698218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 205.570909] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.356989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.439822] Lustre: 6508:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 217.462593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 217.469651] Lustre: Skipped 1 previous similar message [ 217.536339] Lustre: lustre-MDT0001: new disk, initializing [ 217.588188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.607339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 217.613851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 221.141480] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.650880] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 234.894701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.208713] Lustre: lustre-OST0000: new disk, initializing [ 235.212442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 235.222399] Lustre: 8411:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.316993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 241.550871] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.737023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.746591] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 242.813447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 254.309039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.438833] Lustre: lustre-OST0001: new disk, initializing [ 254.443295] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 254.447729] Lustre: 9467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.515755] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 259.271961] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.643414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 261.650360] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 261.680729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 270.376463] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 276.804290] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 282.665441] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing check_logdir /tmp/testlogs/ [ 287.215659] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing yml_node [ 291.193822] Lustre: DEBUG MARKER: Client: 2.17.53.3 [ 293.290230] Lustre: DEBUG MARKER: MDS: 2.17.53.3 [ 295.249216] Lustre: DEBUG MARKER: OSS: 2.17.53.3 [ 296.183941] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed May 13 02:06:45 EDT 2026 [ 313.492689] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 314.809370] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 330.209574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 330.212561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.233950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.285410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.286556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.299884] Lustre: Skipped 1 previous similar message [ 333.314532] Lustre: Skipped 3 previous similar messages [ 335.554142] Lustre: server umount lustre-MDT0000 complete [ 338.858438] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778652449 with bad export cookie 13339234132169701804 [ 338.877082] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 343.520352] LustreError: 6513:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.520596] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.532036] LustreError: 6513:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 343.550649] Lustre: Skipped 1 previous similar message [ 343.551115] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.639836] LustreError: 6514:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.646096] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.658676] LustreError: 6514:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 348.682210] Lustre: Skipped 2 previous similar messages [ 349.666193] LustreError: 6513:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.689102] LustreError: 6513:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 353.247159] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 353.505375] Lustre: server umount lustre-MDT0001 complete [ 357.438766] Lustre: server umount lustre-OST0000 complete [ 361.233433] Lustre: server umount lustre-OST0001 complete [ 369.190465] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 381.541480] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 421.179154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.450447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 421.550727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 421.669875] Lustre: lustre-MDT0000: new disk, initializing [ 421.879547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.921840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 426.512282] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 438.448260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.575117] Lustre: 14985:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 438.677896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 438.685162] Lustre: Skipped 1 previous similar message [ 438.765296] Lustre: lustre-MDT0001: new disk, initializing [ 438.923065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 438.966191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 438.976982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 442.816586] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 449.641256] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 453.119977] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 459.472635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 459.771964] Lustre: lustre-OST0000: new disk, initializing [ 459.776290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 459.783956] Lustre: 16724:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 459.874487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 461.728926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 461.758987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 461.871475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 465.923125] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 472.861534] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 477.025511] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 477.196342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 481.303566] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 481.513925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 487.394946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.400859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 487.406639] Lustre: Skipped 1 previous similar message [ 487.414678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 487.434452] Lustre: Skipped 3 previous similar messages [ 489.322952] Lustre: server umount lustre-OST0000 complete [ 492.512661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.521313] Lustre: Skipped 2 previous similar messages [ 497.636475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 497.643217] Lustre: Skipped 3 previous similar messages [ 498.816242] Lustre: server umount lustre-MDT0000 complete [ 501.582316] LustreError: 15771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778652612 with bad export cookie 13339234132169703330 [ 501.586717] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 501.596548] LustreError: 15771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 501.787842] Lustre: server umount lustre-MDT0001 complete [ 504.144705] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 02:10:13 (1778652613) [ 509.905287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.250716] LustreError: 18840:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.301958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.841880] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 515.555413] LustreError: 18841:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.553508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.020503] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 528.130742] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 530.561591] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 535.492359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 540.293181] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 544.556729] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 560.610896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.622912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 560.623210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 560.637198] Lustre: Skipped 1 previous similar message [ 562.332451] Lustre: server umount lustre-OST0000 complete [ 571.472132] Lustre: server umount lustre-MDT0000 complete [ 574.619312] LustreError: 18820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778652685 with bad export cookie 13339234132169704219 [ 574.623932] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.628793] LustreError: 18820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 574.926841] Lustre: server umount lustre-MDT0001 complete [ 581.786575] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 583.852936] Key type lgssc unregistered [ 584.072673] LNet: 22035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.080975] LNetError: 22035:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 584.094257] LNet: Removed LNI 192.168.201.123@tcp [ 584.735416] Key type .llcrypt unregistered [ 584.737228] Key type ._llcrypt unregistered [ 600.790483] Key type ._llcrypt registered [ 600.792427] Key type .llcrypt registered [ 600.856623] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 601.560611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 601.620582] alg: No test for adler32 (adler32-zlib) [ 602.588488] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 602.728492] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 604.352303] Key type lgssc registered [ 605.150084] Lustre: Echo OBD driver; http://www.lustre.org/ [ 610.552426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 610.571088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.890101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.262937] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 619.974509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.265061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 622.903232] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 626.537372] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 628.841635] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 633.600613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 633.808519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 636.919652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 637.947503] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 642.156549] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 655.331716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.333939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 655.342605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 655.357819] Lustre: Skipped 1 previous similar message [ 660.180573] Lustre: server umount lustre-OST0000 complete [ 665.568957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.571483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 665.584835] Lustre: Skipped 2 previous similar messages [ 665.597981] Lustre: Skipped 1 previous similar message [ 668.903764] Lustre: server umount lustre-MDT0000 complete [ 670.687813] LustreError: 23409:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.699637] LustreError: 23409:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 671.246470] LustreError: 23389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778652782 with bad export cookie 8116957407086823552 [ 671.252703] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 671.255475] LustreError: 23389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 671.434766] Lustre: server umount lustre-MDT0001 complete [ 678.275527] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 680.111418] Key type lgssc unregistered [ 680.320658] LNet: 27025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.327363] LNetError: 27025:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 680.348986] LNet: Removed LNI 192.168.201.123@tcp [ 680.918562] Key type .llcrypt unregistered [ 680.922228] Key type ._llcrypt unregistered [ 697.074958] Key type ._llcrypt registered [ 697.076622] Key type .llcrypt registered [ 697.160848] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 697.882856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 697.896597] alg: No test for adler32 (adler32-zlib) [ 698.844509] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 699.000851] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 700.639255] Key type lgssc registered [ 701.499878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 706.496429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 706.510108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.809569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 710.394356] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 716.810503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 717.150596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 720.126424] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 723.478133] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 725.911586] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 734.754578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 734.990256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 738.490421] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 741.839198] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 753.458626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 762.848422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 762.854048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.862396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 763.876731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 763.876952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 763.878907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 763.898617] Lustre: Skipped 1 previous similar message [ 766.981868] Lustre: server umount lustre-OST0000 complete [ 769.399905] Lustre: server umount lustre-MDT0000 complete [ 771.919264] LustreError: 28380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778652882 with bad export cookie 10887617303283576932 [ 771.922060] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 771.926925] LustreError: 28380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 778.188414] Lustre: server umount lustre-MDT0001 complete [ 784.754188] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 786.361977] Key type lgssc unregistered [ 786.548574] LNet: 31941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.553306] LNetError: 31941:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.627656] LNet: Removed LNI 192.168.201.123@tcp [ 788.096708] Key type .llcrypt unregistered [ 788.099375] Key type ._llcrypt unregistered [ 801.747183] Key type ._llcrypt registered [ 801.749879] Key type .llcrypt registered [ 801.809560] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 802.421929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 802.447658] alg: No test for adler32 (adler32-zlib) [ 803.342813] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 803.467395] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 805.063187] Key type lgssc registered [ 805.560575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 810.476197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 810.490253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.759383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 814.447846] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 819.497954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.786340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 822.071280] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 825.532734] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 827.553924] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 831.264847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 831.496156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 834.645925] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 836.593803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 837.591787] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 849.894301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 849.898339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 849.909908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 849.913096] Lustre: Skipped 1 previous similar message [ 853.440405] Lustre: server umount lustre-OST0000 complete [ 860.128412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 860.132203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 860.137988] Lustre: Skipped 1 previous similar message [ 860.140693] Lustre: Skipped 1 previous similar message [ 865.250852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 865.254093] Lustre: Skipped 2 previous similar messages [ 870.368289] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 870.370029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 870.397509] Lustre: Skipped 1 previous similar message [ 870.568745] Lustre: server umount lustre-MDT0000 complete [ 874.725964] LustreError: 33296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778652985 with bad export cookie 15158265556598311286 [ 874.730881] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 874.734287] LustreError: 33296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 875.025968] Lustre: server umount lustre-MDT0001 complete [ 885.734895] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 888.277908] Key type lgssc unregistered [ 888.555818] LNet: 36488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.565681] LNetError: 36488:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.574685] LNet: Removed LNI 192.168.201.123@tcp [ 889.361667] Key type .llcrypt unregistered [ 889.363374] Key type ._llcrypt unregistered [ 905.658306] Key type ._llcrypt registered [ 905.659698] Key type .llcrypt registered [ 905.725737] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 906.403110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 906.487274] alg: No test for adler32 (adler32-zlib) [ 907.428262] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 907.571749] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 909.207132] Key type lgssc registered [ 909.940356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 914.971105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 914.987609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.284192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.187888] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 926.705727] LustreError: 37863:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.731685] LustreError: 37863:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 930.460195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.202553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 937.409768] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 944.816817] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 950.179448] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 959.927214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 960.634870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 961.677112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 970.822623] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 979.645673] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 987.620972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 987.632877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.649373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.715280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.721513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 991.743861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.477252] Lustre: server umount lustre-OST0000 complete [ 1016.290674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.302623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1016.317607] Lustre: Skipped 3 previous similar messages [ 1020.262166] Lustre: server umount lustre-MDT0000 complete [ 1021.414665] LustreError: 37863:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1021.450733] LustreError: 37863:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1024.037867] LustreError: 37845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778653135 with bad export cookie 10035160024338744479 [ 1024.052548] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1030.524737] Lustre: server umount lustre-MDT0001 complete [ 1042.215212] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1045.185865] Key type lgssc unregistered [ 1045.533073] LNet: 41220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.544991] LNetError: 41220:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.563716] LNet: Removed LNI 192.168.201.123@tcp [ 1046.747282] Key type .llcrypt unregistered [ 1046.749927] Key type ._llcrypt unregistered [ 1069.371538] Key type ._llcrypt registered [ 1069.374165] Key type .llcrypt registered [ 1069.513935] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1070.903604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1070.919543] alg: No test for adler32 (adler32-zlib) [ 1071.961341] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 1072.143090] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1073.800048] Key type lgssc registered [ 1074.951611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1082.395156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1082.428457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1083.973494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1088.038091] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1093.154032] LustreError: 42614:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1093.185082] LustreError: 42614:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1097.597311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.885986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1101.671943] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1107.405141] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1111.085626] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1118.514033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1118.890918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1124.609667] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1128.153533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1131.183489] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1138.471886] Lustre: server umount lustre-MDT0000 complete [ 1138.495326] LustreError: 42610:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.656786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.677154] Lustre: Skipped 2 previous similar messages [ 1138.680971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1142.302966] LustreError: 42593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778653253 with bad export cookie 11271306485586856247 [ 1142.311659] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1142.316379] LustreError: 42593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1143.545284] LustreError: 42610:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1143.576108] LustreError: 42610:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1143.580816] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.23@tcp (stopping) [ 1143.777898] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.648637] Lustre: server umount lustre-MDT0001 complete [ 1165.977203] Lustre: server umount lustre-OST0000 complete [ 1186.371218] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1189.828112] Key type lgssc unregistered [ 1190.233458] LNet: 46113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.241719] LNetError: 46113:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.252709] LNet: Removed LNI 192.168.201.123@tcp [ 1191.313839] Key type .llcrypt unregistered [ 1191.320459] Key type ._llcrypt unregistered [ 1218.621485] Key type ._llcrypt registered [ 1218.625966] Key type .llcrypt registered [ 1218.785709] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1221.141279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1221.211098] alg: No test for adler32 (adler32-zlib) [ 1222.527066] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 1222.752864] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1224.407186] Key type lgssc registered [ 1225.996444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1235.347198] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1235.366591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1251.041294] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1256.159314] LustreError: 47484:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 1282.975593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.084309] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1299.991364] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1465.397761] Lustre: server umount lustre-OST0000 complete [ 1481.713295] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1484.328867] Key type lgssc unregistered [ 1484.591563] LNet: 49032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.602830] LNetError: 49032:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1485.674105] LNet: Removed LNI 192.168.201.123@tcp [ 1486.481193] Key type .llcrypt unregistered [ 1486.484874] Key type ._llcrypt unregistered [ 1509.247770] Key type ._llcrypt registered [ 1509.250627] Key type .llcrypt registered [ 1509.368867] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1510.946337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1510.992095] alg: No test for adler32 (adler32-zlib) [ 1512.147668] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 1512.402448] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1514.055199] Key type lgssc registered [ 1515.171731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1523.194167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1523.239941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1524.854681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1529.442812] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1534.951064] LustreError: 50428:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.973601] LustreError: 50428:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1539.642902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1540.071786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1544.379397] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1550.015623] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1553.566990] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1559.311732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1559.610879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1565.039639] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1569.853865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1570.165445] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1574.882686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1574.896830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.911038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1574.923182] Lustre: Skipped 1 previous similar message [ 1578.521477] Lustre: server umount lustre-OST0000 complete [ 1585.121469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1585.134408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1585.142350] Lustre: Skipped 2 previous similar messages [ 1590.240850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1590.243821] Lustre: Skipped 1 previous similar message [ 1595.361313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1595.370043] Lustre: Skipped 1 previous similar message [ 1596.383127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1596.550814] Lustre: server umount lustre-MDT0000 complete [ 1600.473790] LustreError: 50410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778653711 with bad export cookie 3900895204431935101 [ 1600.481176] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1600.490743] LustreError: 50410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1600.510830] LustreError: 49737:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006e2624f1 x1865053501418496/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1600.855264] Lustre: server umount lustre-MDT0001 complete [ 1609.936974] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1612.531976] Key type lgssc unregistered [ 1612.837493] LNet: 53600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.841865] LNetError: 53600:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.853719] LNet: Removed LNI 192.168.201.123@tcp [ 1613.496677] Key type .llcrypt unregistered [ 1613.498295] Key type ._llcrypt unregistered [ 1632.166225] Key type ._llcrypt registered [ 1632.169404] Key type .llcrypt registered [ 1632.264044] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1633.421032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1633.442084] alg: No test for adler32 (adler32-zlib) [ 1634.591583] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 1634.797503] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1636.447899] Key type lgssc registered [ 1637.359785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1643.888243] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1643.902528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1659.488981] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1664.607219] LustreError: 54950:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 1691.295180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1696.799316] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1702.237573] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1708.571388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1708.962627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.019075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1712.551946] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1714.150133] LustreError: 54976:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.168752] LustreError: 54976:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1719.275230] LustreError: 54977:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1719.820740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.164892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1723.787899] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1728.741054] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1732.544518] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1734.624805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1734.637154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.657047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1735.653281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1735.681612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1735.712191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1740.596675] Lustre: server umount lustre-OST0000 complete [ 1760.328433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.329570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1760.340604] Lustre: Skipped 1 previous similar message [ 1760.355113] Lustre: Skipped 3 previous similar messages [ 1766.744509] Lustre: server umount lustre-MDT0000 complete [ 1766.881656] LustreError: 55873:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1766.894986] LustreError: 55873:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1770.245502] LustreError: 54957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778653881 with bad export cookie 11100824647450695331 [ 1770.252767] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1776.608791] Lustre: server umount lustre-MDT0001 complete [ 1786.891504] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1790.244032] Key type lgssc unregistered [ 1790.608079] LNet: 58338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.623500] LNetError: 58338:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1790.661411] LNet: Removed LNI 192.168.201.123@tcp [ 1791.670684] Key type .llcrypt unregistered [ 1791.675920] Key type ._llcrypt unregistered [ 1813.434821] Key type ._llcrypt registered [ 1813.436565] Key type .llcrypt registered [ 1813.536858] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1814.239915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1814.394579] alg: No test for adler32 (adler32-zlib) [ 1815.439524] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 1815.697987] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1817.447311] Key type lgssc registered [ 1818.994046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1826.302713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1826.322625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.731815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1831.926404] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1837.032853] LustreError: 59735:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.043070] LustreError: 59735:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1840.463334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.829481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1844.407414] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1848.654617] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1851.783367] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1857.697899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1858.076586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.368481] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1866.615113] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1872.437315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1882.592700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1882.595118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.614910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1886.835802] Lustre: server umount lustre-OST0000 complete [ 1892.323075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1892.327711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.336542] Lustre: Skipped 1 previous similar message [ 1892.342341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1892.348568] Lustre: Skipped 1 previous similar message [ 1897.953291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1897.964518] Lustre: Skipped 2 previous similar messages [ 1902.050734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1904.095166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1904.307092] Lustre: server umount lustre-MDT0000 complete [ 1907.169688] LustreError: 59735:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1907.195237] LustreError: 59735:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1907.530984] LustreError: 59716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778654018 with bad export cookie 13853203793601398776 [ 1907.546745] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1907.756402] Lustre: server umount lustre-MDT0001 complete [ 1916.471364] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1918.746531] Key type lgssc unregistered [ 1918.956436] LNet: 62930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1918.959651] LNetError: 62930:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1918.976392] LNet: Removed LNI 192.168.201.123@tcp [ 1919.720104] Key type .llcrypt unregistered [ 1919.722103] Key type ._llcrypt unregistered [ 1939.017373] Key type ._llcrypt registered [ 1939.018733] Key type .llcrypt registered [ 1939.111972] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1940.009148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1940.025124] alg: No test for adler32 (adler32-zlib) [ 1940.926981] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 1941.041216] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1942.647183] Key type lgssc registered [ 1943.512789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1950.725308] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 02:34:20 (1778654060) [ 1956.037115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1956.050207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.386193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1960.589436] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1966.924727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.194899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1969.944486] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1974.091647] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1977.137138] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1982.128969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1982.399500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.404385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1986.495180] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1989.965549] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1999.022605] Lustre: Failing over lustre-MDT0000 [ 1999.224723] Lustre: server umount lustre-MDT0000 complete [ 2002.702077] LustreError: 64593:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.714120] LustreError: 64593:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2002.914171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2002.917822] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.937861] Lustre: Skipped 1 previous similar message [ 2005.517205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2005.601392] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2005.683779] Lustre: *** cfs_fail_loc=135, val=0*** [ 2005.740630] LustreError: 67342:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2005.744553] LustreError: 67342:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.123@tcp: cfg command failed: rc = -2 [ 2005.748349] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2005.748349] [ 2005.752642] LustreError: MGC192.168.201.123@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2005.759264] LustreError: 67331:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2005.762594] LustreError: 67331:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2005.768534] LustreError: 67331:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2005.773083] LustreError: 67331:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2005.789793] Lustre: server umount lustre-MDT0000 complete [ 2005.791858] LustreError: 67331:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2007.801659] LustreError: 64593:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.818463] LustreError: 64593:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2010.153470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.225546] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2010.363398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.386824] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2012.456941] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2012.925062] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2015.725486] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2015.742189] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2015.795937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2018.274405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2018.280857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.288869] Lustre: Skipped 1 previous similar message [ 2018.295183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2020.834213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2020.836606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2024.100554] Lustre: server umount lustre-OST0000 complete [ 2031.075537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.078278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.083859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2031.086203] Lustre: Skipped 1 previous similar message [ 2031.097315] Lustre: Skipped 2 previous similar messages [ 2033.763389] Lustre: server umount lustre-MDT0000 complete [ 2036.195374] LustreError: 65386:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.200831] LustreError: 65386:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 2036.900539] LustreError: 64578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778654147 with bad export cookie 2296854994928734093 [ 2036.903511] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2036.913131] LustreError: 64578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2037.272572] Lustre: server umount lustre-MDT0001 complete [ 2045.989697] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2048.127867] Key type lgssc unregistered [ 2048.355294] LNet: 69085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2048.359383] LNetError: 69085:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2048.367483] LNet: Removed LNI 192.168.201.123@tcp [ 2049.029223] Key type .llcrypt unregistered [ 2049.031606] Key type ._llcrypt unregistered [ 2067.312736] Key type ._llcrypt registered [ 2067.314734] Key type .llcrypt registered [ 2067.402282] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2068.143282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2068.238191] alg: No test for adler32 (adler32-zlib) [ 2069.196900] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 2069.356085] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2070.991273] Key type lgssc registered [ 2072.111643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2078.453225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2078.472966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.160458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.779374] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2090.479585] LustreError: 70461:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.534256] LustreError: 70461:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2095.591248] LustreError: 70462:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2098.768384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2099.341888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2105.277169] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2113.918158] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2120.062789] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2129.760835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2130.382412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.740736] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2148.125657] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2150.859909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2159.128716] Lustre: Failing over lustre-MDT0000 [ 2159.684741] Lustre: server umount lustre-MDT0000 complete [ 2159.903884] LustreError: 71414:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.612720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2160.616288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.196569] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.199597] LustreError: 70462:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.256392] LustreError: 70462:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2169.328637] LustreError: 70456:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.344874] LustreError: 70456:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2171.146027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.396650] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2171.621797] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2171.778072] LustreError: 73213:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2171.793218] LustreError: 73213:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.123@tcp: cfg command failed: rc = -9 [ 2171.798227] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2171.798227] [ 2171.816883] LustreError: MGC192.168.201.123@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2171.840415] LustreError: 73201:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2171.846338] LustreError: 73201:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 2171.854256] LustreError: 73201:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2171.861252] LustreError: 73201:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2171.905412] Lustre: server umount lustre-MDT0000 complete [ 2171.908991] LustreError: 73201:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2178.766728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.886661] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2179.291656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2179.398870] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2181.369849] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2184.222629] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2184.688361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2184.744528] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2184.821136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2194.914284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.933224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2194.937759] Lustre: Skipped 2 previous similar messages [ 2198.347564] Lustre: server umount lustre-OST0000 complete [ 2205.167430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.169086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2205.185595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2205.186050] Lustre: Skipped 1 previous similar message [ 2205.190345] Lustre: Skipped 1 previous similar message [ 2208.924903] Lustre: server umount lustre-MDT0000 complete [ 2210.276117] LustreError: 70461:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.303271] LustreError: 70461:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 2212.481792] LustreError: 70443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778654323 with bad export cookie 14050458501885300376 [ 2212.494036] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2212.868719] Lustre: server umount lustre-MDT0001 complete [ 2222.477727] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2225.377173] Key type lgssc unregistered [ 2225.746833] LNet: 74960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2225.754344] LNetError: 74960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2225.788761] LNet: Removed LNI 192.168.201.123@tcp [ 2226.600665] Key type .llcrypt unregistered [ 2226.603796] Key type ._llcrypt unregistered [ 2239.902380] Key type ._llcrypt registered [ 2239.904983] Key type .llcrypt registered [ 2240.056053] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2240.521334] Key type .llcrypt unregistered [ 2240.523949] Key type ._llcrypt unregistered [ 2267.667739] Key type ._llcrypt registered [ 2267.673284] Key type .llcrypt registered [ 2267.797537] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2268.846489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2268.900068] alg: No test for adler32 (adler32-zlib) [ 2269.951895] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 2270.268630] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2271.983168] Key type lgssc registered [ 2273.305441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2282.366769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2282.381698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2283.950547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.019045] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2293.226239] LustreError: 76828:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.258520] LustreError: 76828:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2298.113483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.339686] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2298.558388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2302.336671] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2309.029604] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2314.131158] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2321.915321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2322.397299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2328.588695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2330.065545] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2337.840377] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2354.144526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2354.152392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.160601] LustreError: Skipped 1 previous similar message [ 2354.178254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2354.202360] Lustre: Skipped 1 previous similar message [ 2355.620610] Lustre: server umount lustre-OST0000 complete [ 2360.291055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2360.306325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.326179] Lustre: Skipped 1 previous similar message [ 2360.331888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2364.384833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.385077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2364.406532] Lustre: Skipped 1 previous similar message [ 2366.713738] Lustre: server umount lustre-MDT0000 complete [ 2369.511345] LustreError: 76823:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.554566] LustreError: 76823:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2371.055820] LustreError: 76810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778654482 with bad export cookie 9277833294434453104 [ 2371.074855] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2371.078050] LustreError: 76810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2371.859429] Lustre: server umount lustre-MDT0001 complete [ 2386.598549] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2390.965883] Key type lgssc unregistered [ 2391.488873] LNet: 80008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2391.495724] LNetError: 80008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2391.507655] LNet: Removed LNI 192.168.201.123@tcp [ 2392.171215] Key type .llcrypt unregistered [ 2392.172982] Key type ._llcrypt unregistered [ 2419.611227] Key type ._llcrypt registered [ 2419.614174] Key type .llcrypt registered [ 2419.794758] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2421.471395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2421.613367] alg: No test for adler32 (adler32-zlib) [ 2422.844941] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 2423.247153] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2425.031199] Key type lgssc registered [ 2426.467066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2435.504534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2435.573712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2437.384225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.665362] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2447.846181] LustreError: 81406:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.878888] LustreError: 81406:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2452.971820] LustreError: 81405:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.148410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.584153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2458.222106] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2465.511812] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2469.725354] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2476.845425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2477.378543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.484208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2484.303128] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2490.330191] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2498.017295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2498.026092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.036810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2498.046547] Lustre: Skipped 1 previous similar message [ 2499.205288] Lustre: server umount lustre-OST0000 complete [ 2504.672740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2504.687824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.718670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2504.721061] Lustre: Skipped 1 previous similar message [ 2508.257621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2508.264632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.949682] Lustre: server umount lustre-MDT0000 complete [ 2513.378390] LustreError: 81405:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.405129] LustreError: 81405:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2514.052559] LustreError: 83023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778654625 with bad export cookie 3316304099033667767 [ 2514.057779] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2514.062961] LustreError: 83023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2514.488894] Lustre: server umount lustre-MDT0001 complete [ 2529.133756] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2532.606617] Key type lgssc unregistered [ 2532.972954] LNet: 84582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2532.979686] LNetError: 84582:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2532.997325] LNet: Removed LNI 192.168.201.123@tcp [ 2533.691893] Key type .llcrypt unregistered [ 2533.693629] Key type ._llcrypt unregistered [ 2563.940651] Key type ._llcrypt registered [ 2563.943487] Key type .llcrypt registered [ 2564.077959] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2565.323133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2565.425597] alg: No test for adler32 (adler32-zlib) [ 2566.612501] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 2566.803963] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2568.463408] Key type lgssc registered [ 2569.931783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2578.629319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2578.644286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2580.069730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2583.895590] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2590.184990] LustreError: 85981:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.215467] LustreError: 85981:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2592.022904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2592.394735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2596.932709] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2602.442709] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2606.606746] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2613.721825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2614.097188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2621.345519] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2625.397705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2627.095596] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2640.592084] Lustre: server umount lustre-OST0000 complete [ 2645.985665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2645.988284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2645.998787] Lustre: Skipped 1 previous similar message [ 2651.108095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2651.110991] Lustre: Skipped 2 previous similar messages [ 2655.347864] Lustre: server umount lustre-MDT0000 complete [ 2656.223168] Lustre: 85291:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778654751/real 1778654751] req@00000000c0c439c4 x1865054606617216/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1778654767 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2656.224486] LustreError: 85982:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.254462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2656.305701] LustreError: 85982:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2658.271466] Lustre: 85292:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778654753/real 1778654753] req@00000000172aa739 x1865054606617728/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1778654769 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2659.169120] LustreError: 85962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778654770 with bad export cookie 3026629502239634019 [ 2659.177797] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2659.179303] LustreError: 85962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2659.610770] Lustre: server umount lustre-MDT0001 complete [ 2671.246821] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2673.793793] Key type lgssc unregistered [ 2674.058378] LNet: 89156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2674.066536] LNetError: 89156:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2674.078313] LNet: Removed LNI 192.168.201.123@tcp [ 2674.972658] Key type .llcrypt unregistered [ 2674.974748] Key type ._llcrypt unregistered [ 2694.832989] Key type ._llcrypt registered [ 2694.835566] Key type .llcrypt registered [ 2694.928563] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2695.970984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2695.994157] alg: No test for adler32 (adler32-zlib) [ 2697.106079] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 2697.272783] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2698.935187] Key type lgssc registered [ 2700.012366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2706.239679] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2706.250222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2721.763168] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2726.879201] LustreError: 90508:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 2753.567227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2759.564632] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2765.075628] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2771.217054] Lustre: server umount lustre-OST0000 complete [ 2779.693490] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 02:48:08 (1778654888) [ 2787.177264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2787.949709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2792.616888] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2793.447389] LustreError: 92032:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2793.482313] LustreError: 92032:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2797.550632] LustreError: 92033:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.878860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2801.340261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2805.424505] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2810.698724] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2814.954695] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2821.533084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2821.885253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2827.697438] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2832.119364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2833.486853] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2846.694852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2846.712399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.740264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2847.728961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2847.751281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.765757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2852.500487] Lustre: server umount lustre-OST0000 complete [ 2863.087894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2863.095710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2863.115162] Lustre: Skipped 1 previous similar message [ 2863.116969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2863.135988] Lustre: Skipped 2 previous similar messages [ 2868.192821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.208380] Lustre: Skipped 1 previous similar message [ 2872.289319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2872.294063] Lustre: Skipped 1 previous similar message [ 2873.311108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2873.540106] Lustre: server umount lustre-MDT0000 complete [ 2877.407944] LustreError: 92744:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.435035] LustreError: 92744:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2877.839298] LustreError: 92013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778654988 with bad export cookie 17203710661963053852 [ 2877.849451] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2878.150576] Lustre: server umount lustre-MDT0001 complete [ 2889.460663] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2892.298894] Key type lgssc unregistered [ 2892.663985] LNet: 95370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2892.671660] LNetError: 95370:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2893.735197] LNet: Removed LNI 192.168.201.123@tcp [ 2894.491313] Key type .llcrypt unregistered [ 2894.495646] Key type ._llcrypt unregistered [ 2943.475617] Key type ._llcrypt registered [ 2943.479363] Key type .llcrypt registered [ 2943.770567] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2945.066280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2945.238755] alg: No test for adler32 (adler32-zlib) [ 2946.715815] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 2947.104231] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2948.767207] Key type lgssc registered [ 2950.739447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2960.983312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2961.001163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.367185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2971.372746] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2973.668551] LustreError: 97543:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2973.711285] LustreError: 97543:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2978.800353] LustreError: 97542:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2983.913421] LustreError: 97543:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2986.767051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2987.394156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2992.449866] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2999.814139] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3004.799731] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3013.202408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3013.620929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3021.074296] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3032.414550] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3033.096667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3043.303241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3043.309201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3043.323589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3048.427776] LustreError: 99695:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3048.441940] LustreError: 99695:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3048.638558] Lustre: server umount lustre-OST0000 complete [ 3053.538526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3053.541944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3053.556256] Lustre: Skipped 2 previous similar messages [ 3053.562958] Lustre: Skipped 1 previous similar message [ 3058.656309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3058.665083] Lustre: Skipped 2 previous similar messages [ 3058.949677] Lustre: server umount lustre-MDT0000 complete [ 3063.017780] LustreError: 99785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778655174 with bad export cookie 1802474161726642656 [ 3063.033046] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3063.045436] LustreError: 99785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3063.624928] Lustre: server umount lustre-MDT0001 complete [ 3077.123434] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 3081.034829] Key type lgssc unregistered [ 3081.534051] LNet: 100723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3081.553946] LNetError: 100723:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3081.603033] LNet: Removed LNI 192.168.201.123@tcp [ 3083.081604] Key type .llcrypt unregistered [ 3083.084599] Key type ._llcrypt unregistered [ 3102.876425] Key type ._llcrypt registered [ 3102.878399] Key type .llcrypt registered [ 3102.990582] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3104.116128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3104.131266] alg: No test for adler32 (adler32-zlib) [ 3105.130626] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 3105.334142] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 3106.983855] Key type lgssc registered [ 3108.155206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3117.373320] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3117.391139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3133.087304] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3138.207179] LustreError: 101951:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 3164.895411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3171.038169] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3176.634429] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3183.775932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.215966] LustreError: MGC192.168.201.123@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3184.235667] LustreError: 102862:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3184.241881] LustreError: 102862:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3184.251651] LustreError: 102862:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3184.336404] Lustre: server umount lustre-MDT0000 complete [ 3184.338936] LustreError: 102862:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3201.136588] Lustre: server umount lustre-OST0000 complete [ 3211.197726] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 3221.101589] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3259.173478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.515582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3259.591964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3259.753269] Lustre: lustre-MDT0000: new disk, initializing [ 3259.830368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3259.848680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3264.906145] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3278.133322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3278.282432] Lustre: 105620:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3278.487298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3278.492845] Lustre: Skipped 1 previous similar message [ 3278.626923] Lustre: lustre-MDT0001: new disk, initializing [ 3278.793903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3278.854761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3278.864216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3284.291434] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3294.507380] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3299.077288] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3307.444471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3307.873699] Lustre: lustre-OST0000: new disk, initializing [ 3307.880748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3307.899995] Lustre: 107363:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3308.063139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.038830] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3317.397916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3317.422347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3317.614605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3325.965372] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3331.515257] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3331.869736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3336.160493] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3336.648695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3343.333359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3343.335095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.351338] LustreError: Skipped 1 previous similar message [ 3343.352194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3343.374078] Lustre: Skipped 1 previous similar message [ 3344.907834] Lustre: server umount lustre-OST0000 complete [ 3353.584541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.586465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3353.619353] Lustre: Skipped 1 previous similar message [ 3353.627308] Lustre: Skipped 1 previous similar message [ 3358.153392] Lustre: server umount lustre-MDT0000 complete [ 3362.369440] LustreError: 107362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778655473 with bad export cookie 3643597889728515336 [ 3362.384662] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3362.394543] LustreError: 107362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3362.784442] Lustre: server umount lustre-MDT0001 complete [ 3372.787701] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 02:58:00 (1778655480) [ 3376.276789] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3394.201594] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 3407.111225] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3448.486463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3448.815442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3448.852578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3448.934262] Lustre: lustre-MDT0000: new disk, initializing [ 3449.086633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3449.101622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3454.626520] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3469.182757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.328446] Lustre: 112085:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3469.416174] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3469.425355] Lustre: Skipped 1 previous similar message [ 3469.520287] Lustre: lustre-MDT0001: new disk, initializing [ 3469.677486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3469.856615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3469.869289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3475.113530] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3484.202785] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3488.174626] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3496.048545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3496.451968] Lustre: lustre-OST0000: new disk, initializing [ 3496.454549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3496.463204] Lustre: 113828:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3496.557591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3504.051465] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3506.315699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3506.340451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3506.522518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3514.786979] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3522.511661] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3522.787328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3527.190254] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3527.434432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3532.265151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.267313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3532.287382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3532.287398] Lustre: Skipped 1 previous similar message [ 3532.334654] Lustre: Skipped 1 previous similar message [ 3532.444216] LustreError: Skipped 1 previous similar message [ 3535.927446] Lustre: server umount lustre-OST0000 complete [ 3541.472498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3541.476880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3541.486799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3541.490310] Lustre: Skipped 1 previous similar message [ 3547.616754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3547.636137] Lustre: Skipped 3 previous similar messages [ 3554.271887] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3554.668231] Lustre: server umount lustre-MDT0000 complete [ 3557.858128] LustreError: 112092:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3557.869589] LustreError: 112092:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 3558.692547] LustreError: 112077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778655669 with bad export cookie 3643597889728516211 [ 3558.694097] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3558.705907] LustreError: 112077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3559.159541] Lustre: server umount lustre-MDT0001 complete [ 3567.270907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3567.716416] LustreError: 115913:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3567.795476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3571.865525] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3573.216906] LustreError: 115914:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3577.322943] LustreError: 115913:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3581.673855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.595909] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3593.162177] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3596.890886] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3603.343270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3609.826802] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3615.385631] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3620.791927] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3628.513261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3628.518523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3628.524742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3628.524750] Lustre: Skipped 2 previous similar messages [ 3628.528429] Lustre: Skipped 2 previous similar messages [ 3630.143856] Lustre: server umount lustre-OST0000 complete [ 3638.241079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3638.255143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.744810] Lustre: server umount lustre-MDT0000 complete [ 3643.881584] LustreError: 115914:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3643.911046] LustreError: 115914:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3644.780273] LustreError: 115895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778655755 with bad export cookie 3643597889728517100 [ 3644.796217] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3645.239497] Lustre: server umount lustre-MDT0001 complete [ 3655.617455] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 3658.201679] Key type lgssc unregistered [ 3658.534822] LNet: 119182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3658.548055] LNetError: 119182:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3658.574617] LNet: Removed LNI 192.168.201.123@tcp [ 3659.483432] Key type .llcrypt unregistered [ 3659.484639] Key type ._llcrypt unregistered [ 3677.064444] Key type ._llcrypt registered [ 3677.066341] Key type .llcrypt registered [ 3677.157412] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 3690.730747] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3691.936692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3692.002248] alg: No test for adler32 (adler32-zlib) [ 3693.190942] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 3693.529210] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 3695.271168] Key type lgssc registered [ 3696.720776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3734.437278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3734.483119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.808729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3735.826177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3735.890119] Lustre: lustre-MDT0000: new disk, initializing [ 3735.961462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.975883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3740.071807] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3755.179948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3755.313296] Lustre: 122281:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3755.388730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3755.395691] Lustre: Skipped 1 previous similar message [ 3755.473345] Lustre: lustre-MDT0001: new disk, initializing [ 3755.528142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3755.550429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3755.559141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3761.514512] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3770.901595] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3776.028574] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3783.530805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3783.768344] Lustre: lustre-OST0000: new disk, initializing [ 3783.771662] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3783.783712] Lustre: 124023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3783.886906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3789.120945] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3790.506466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3790.512498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3790.704962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3796.682449] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3800.065203] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3800.293038] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3803.922776] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3804.123854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3806.180104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3806.182581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3806.184536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3806.190495] Lustre: Skipped 1 previous similar message [ 3811.307894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3811.316839] Lustre: Skipped 2 previous similar messages [ 3812.226290] Lustre: server umount lustre-OST0000 complete [ 3816.419648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3816.432671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3816.436946] Lustre: Skipped 1 previous similar message [ 3821.537829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3821.547320] Lustre: Skipped 1 previous similar message [ 3821.999098] Lustre: server umount lustre-MDT0000 complete [ 3826.440148] LustreError: 122273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778655937 with bad export cookie 13860689953035080212 [ 3826.447575] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3826.461853] LustreError: 122273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3826.819733] Lustre: server umount lustre-MDT0001 complete [ 3836.457476] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 03:05:45 (1778655945) [ 3843.229700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3843.634371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3848.566884] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3848.682697] LustreError: 126287:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.705078] LustreError: 126287:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3853.803501] LustreError: 126286:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3858.307106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3858.717382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3863.342970] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3868.698203] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3872.404858] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3878.368668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3878.371803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3878.375754] Lustre: Skipped 2 previous similar messages [ 3881.099613] Lustre: server umount lustre-MDT0000 complete [ 3883.488363] LustreError: 126286:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3883.518914] LustreError: 126286:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3884.875851] LustreError: 126267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778655995 with bad export cookie 13860689953035081094 [ 3884.888683] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3884.894134] LustreError: 126267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3885.226561] Lustre: server umount lustre-MDT0001 complete [ 3893.867931] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 03:06:42 (1778656002) [ 3900.395342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3915.936114] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3921.058246] LustreError: 128477:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 3946.783226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3952.923401] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3960.869143] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3964.048163] Lustre: server umount lustre-OST0000 complete [ 3974.120791] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 03:08:02 (1778656082) [ 3982.498452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.317737] LustreError: 129810:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3983.531232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3988.967100] LustreError: 129811:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3990.443952] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3998.191203] LustreError: 129810:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3998.242465] LustreError: 129810:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4006.654412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4014.182123] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4025.687832] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4029.713936] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4037.697788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4038.306763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.311061] Lustre: Skipped 1 previous similar message [ 4045.981824] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4054.161940] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4064.225057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4064.227671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4064.244984] Lustre: Skipped 2 previous similar messages [ 4064.259182] Lustre: Skipped 3 previous similar messages [ 4071.140754] Lustre: server umount lustre-MDT0000 complete [ 4074.465201] LustreError: 129810:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4074.484954] LustreError: 129810:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4076.758018] LustreError: 129791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778656187 with bad export cookie 13860689953035081759 [ 4076.776546] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4076.785210] LustreError: 129791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4077.256090] Lustre: server umount lustre-MDT0001 complete [ 4092.395560] Lustre: server umount lustre-OST0000 complete [ 4102.746474] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 03:10:11 (1778656211) [ 4111.114386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4111.871322] LustreError: 133128:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.944364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4117.864470] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4128.543507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4136.205860] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4143.621522] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4148.768594] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4155.102811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4156.569830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4161.968764] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4168.571640] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4173.001868] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4173.223560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4178.018942] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4178.178547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4180.454783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4180.457306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4180.474167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4180.474174] Lustre: Skipped 3 previous similar messages [ 4180.498075] Lustre: Skipped 1 previous similar message [ 4182.008965] Lustre: server umount lustre-OST0000 complete [ 4190.690519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4190.702845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4190.740371] Lustre: Skipped 1 previous similar message [ 4192.629984] Lustre: server umount lustre-MDT0000 complete [ 4195.810410] LustreError: 133128:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.825871] LustreError: 133128:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 4196.325955] LustreError: 133110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778656307 with bad export cookie 13860689953035082830 [ 4196.329803] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4196.333487] LustreError: 133110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4196.538318] Lustre: server umount lustre-MDT0001 complete [ 4205.557969] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 03:11:54 (1778656314) [ 4211.869880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4227.423877] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4232.543435] LustreError: 136913:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 4258.335300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4258.347651] Lustre: Skipped 2 previous similar messages [ 4265.162501] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4271.039905] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4278.235650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.360314] LustreError: 120501:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000e8f304b x1865055788460160/t0(0) o250->MGC192.168.201.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4278.910630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4282.555455] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4294.006715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4297.843963] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4303.154530] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4306.603900] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4310.839618] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4311.155694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4316.180479] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4316.503424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4320.226331] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4320.232275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4320.243508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4320.256541] Lustre: Skipped 1 previous similar message [ 4320.296363] Lustre: Skipped 5 previous similar messages [ 4325.182263] Lustre: server umount lustre-MDT0000 complete [ 4325.356064] LustreError: 137838:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.400330] LustreError: 137838:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 4329.716805] LustreError: 136921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778656440 with bad export cookie 13860689953035083852 [ 4329.732344] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4329.737930] LustreError: 136921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4330.214635] Lustre: server umount lustre-MDT0001 complete [ 4345.277132] Lustre: server umount lustre-OST0000 complete [ 4359.156317] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 03:14:26 (1778656466) [ 4369.664884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4385.570323] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4390.687451] LustreError: 140735:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 4423.659994] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4430.228709] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4436.872311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4437.318691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4442.532358] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4451.697437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.866319] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4462.410848] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4466.173378] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4473.614452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4473.768897] Lustre: lustre-OST0001: new disk, initializing [ 4473.772081] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4473.778651] Lustre: 143268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4480.408975] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4484.219145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4484.245772] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4484.406978] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4491.119303] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4499.649777] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4499.988628] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4505.135870] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4505.430325] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4508.643071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4508.649395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4508.668908] Lustre: Skipped 1 previous similar message [ 4508.671062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4513.351301] Lustre: server umount lustre-OST0000 complete [ 4520.417868] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4520.433471] LustreError: Skipped 1 previous similar message [ 4532.191158] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4532.455720] Lustre: server umount lustre-OST0001 complete [ 4539.360540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4543.007349] Lustre: server umount lustre-MDT0000 complete [ 4547.834312] LustreError: 140741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778656658 with bad export cookie 13860689953035084804 [ 4547.850781] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4547.868191] LustreError: 140741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4548.238979] Lustre: server umount lustre-MDT0001 complete [ 4576.790836] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 03:18:05 (1778656685) [ 4578.608669] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4580.044752] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4581.391131] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 03:18:10 (1778656690) [ 4587.304527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4587.767487] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4587.857828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4588.086144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.093593] Lustre: Skipped 6 previous similar messages [ 4592.827500] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4602.334402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4602.402252] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4602.422791] Lustre: 146685:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4607.472206] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4614.180703] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4620.340705] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4629.389447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4629.559045] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4637.000482] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4637.385480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4643.831283] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4649.681944] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4650.049607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4656.083962] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4656.424445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4662.759586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4662.769234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4662.794180] Lustre: Skipped 5 previous similar messages [ 4662.807103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4662.812346] Lustre: Skipped 12 previous similar messages [ 4665.534213] Lustre: server umount lustre-OST0000 complete [ 4688.838906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4695.551893] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4702.330645] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4703.728443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4703.750778] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4708.408882] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4708.768021] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4713.029640] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4713.050232] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4714.937537] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4715.245692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4720.236875] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8cb606eb8000.ost_server_uuid 50 [ 4722.627039] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8cb606eb8000.ost_server_uuid in FULL state after 0 sec [ 4722.902665] LustreError: 146692:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.23@tcp) returned error from blocking AST (req@00000000a95e28d0 x1865055788596992 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000bbcee242/0xc05b1013407b6d5e lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.23@tcp remote: 0xd5217326176b1e3c expref: 5 pid: 148574 timeout: 4822 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4722.945616] LustreError: lustre-MDT0000: A client on nid 192.168.201.23@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4722.963626] LustreError: 146680:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.23@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000bbcee242/0xc05b1013407b6d5e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.23@tcp remote: 0xd5217326176b1e3c expref: 6 pid: 148574 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4734.544524] Lustre: server umount lustre-OST0000 complete [ 4741.096813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4741.116300] LustreError: Skipped 1 previous similar message [ 4748.252083] Lustre: server umount lustre-MDT0000 complete [ 4748.779259] LustreError: 146696:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4748.823771] LustreError: 146696:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 16 previous similar messages [ 4753.737522] LustreError: 146677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778656864 with bad export cookie 13860689953035085987 [ 4753.756343] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4753.759692] LustreError: 146677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4768.226172] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4768.889537] Lustre: server umount lustre-MDT0001 complete [ 4784.619404] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 4789.856596] Key type lgssc unregistered [ 4790.327467] LNet: 152172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4790.332426] LNetError: 152172:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4790.351480] LNet: Removed LNI 192.168.201.123@tcp [ 4791.575718] Key type .llcrypt unregistered [ 4791.577898] Key type ._llcrypt unregistered [ 4822.417764] Key type ._llcrypt registered [ 4822.429124] Key type .llcrypt registered [ 4822.698897] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 4825.031924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4825.209902] alg: No test for adler32 (adler32-zlib) [ 4826.663878] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 4827.025383] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 4828.807697] Key type lgssc registered [ 4830.438507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4840.090548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4840.108590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4841.782233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4847.614733] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4851.171076] LustreError: 153579:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4851.215038] LustreError: 153579:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4856.291530] LustreError: 153578:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4857.650618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4858.132269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4864.729469] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4872.518686] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4877.851311] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4886.010651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4886.611434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4894.518656] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4901.039767] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4911.226149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4920.384287] Lustre: Failing over lustre-MDT0000 [ 4921.013559] Lustre: server umount lustre-MDT0000 complete [ 4921.830652] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4921.849832] LustreError: 153573:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.878505] Lustre: Skipped 1 previous similar message [ 4921.939588] LustreError: 153573:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4926.949160] LustreError: 153578:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4926.978222] LustreError: 153578:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4932.066550] LustreError: 153579:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4932.083227] LustreError: 153579:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4934.222776] LustreError: 153559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.23@tcp arrived at 1778657045 with bad export cookie 13039002660548101132 [ 4934.243735] LustreError: 153559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4937.206773] Lustre: 152886:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778657032/real 1778657032] req@00000000dcf159c5 x1865056977451136/t0(0) o400->MGC192.168.201.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1778657048 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4937.276562] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4942.313784] LustreError: 153574:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4942.332917] LustreError: 153574:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 4943.351708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4947.786343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4947.849368] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4953.075344] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4953.093593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4954.072809] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4956.532691] Lustre: lustre-MDT0000: Denying connection for new client e1e814b8-6869-4916-8919-be167213882d (at 192.168.201.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 4968.425996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4968.436923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4968.453148] Lustre: Skipped 2 previous similar messages [ 4968.472170] Lustre: Skipped 1 previous similar message [ 4973.538942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4973.561967] Lustre: Skipped 1 previous similar message [ 4978.663753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4978.680304] Lustre: Skipped 1 previous similar message [ 4980.191310] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4980.507666] Lustre: server umount lustre-OST0000 complete [ 4985.873917] LustreError: 156975:0:(ldlm_lib.c:2986:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4985.878048] Lustre: 156339:0:(ldlm_lib.c:2389:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4985.883404] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4985.894178] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4985.902087] LustreError: 156339:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e4a3bb3c x1865056977472768/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4985.911840] LustreError: 156339:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4985.917164] LustreError: 156339:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4985.922776] Lustre: lustre-MDT0000: Recovery over after 0:32, of 2 clients 0 recovered and 2 were evicted. [ 4986.399289] Lustre: server umount lustre-MDT0000 complete [ 4988.908458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4988.925187] LustreError: 153573:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4988.943472] LustreError: 153573:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4991.326835] LustreError: 153558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778657102 with bad export cookie 13039002660548101468 [ 4991.330126] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4991.346922] LustreError: 153558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4991.370351] LustreError: 152885:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4991.905739] Lustre: server umount lustre-MDT0001 complete [ 5004.550850] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5007.316545] Key type lgssc unregistered [ 5007.671180] LNet: 157711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5007.675807] LNetError: 157711:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5007.696481] LNet: Removed LNI 192.168.201.123@tcp [ 5008.654830] Key type .llcrypt unregistered [ 5008.660956] Key type ._llcrypt unregistered [ 5035.161991] Key type ._llcrypt registered [ 5035.166444] Key type .llcrypt registered [ 5035.330702] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5036.951547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5037.058790] alg: No test for adler32 (adler32-zlib) [ 5038.179958] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 5038.380146] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 5040.023152] Key type lgssc registered [ 5041.074535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5049.765709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5049.789489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5051.329688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5056.095969] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5061.604263] LustreError: 159107:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5061.630298] LustreError: 159107:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5066.076493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.596613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5072.343552] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5079.534903] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5086.769707] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5096.892854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5097.250143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5103.684052] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5113.036667] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5122.002962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5132.257152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5132.267860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5132.270779] Lustre: Skipped 1 previous similar message [ 5132.291408] Lustre: Skipped 1 previous similar message [ 5136.813795] Lustre: server umount lustre-OST0000 complete [ 5142.500269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5142.504337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5142.531336] Lustre: Skipped 1 previous similar message [ 5147.620460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5147.634557] Lustre: Skipped 2 previous similar messages [ 5148.356947] Lustre: server umount lustre-MDT0000 complete [ 5152.736690] LustreError: 160021:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5152.756330] LustreError: 160021:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5152.922496] LustreError: 159086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778657263 with bad export cookie 11067841796349301070 [ 5152.925316] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5152.930916] LustreError: 159086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5153.270856] Lustre: server umount lustre-MDT0001 complete [ 5165.609435] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5170.358545] Key type lgssc unregistered [ 5170.873505] LNet: 162306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5170.883568] LNetError: 162306:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5170.921322] LNet: Removed LNI 192.168.201.123@tcp [ 5172.221189] Key type .llcrypt unregistered [ 5172.225724] Key type ._llcrypt unregistered [ 5221.962867] Key type ._llcrypt registered [ 5221.968517] Key type .llcrypt registered [ 5222.175683] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5224.257344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5224.520551] alg: No test for adler32 (adler32-zlib) [ 5225.715610] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 5226.266342] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 5228.064091] Key type lgssc registered [ 5229.426745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5239.616487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5239.656997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5241.434644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5246.647227] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5251.564290] LustreError: 163995:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5251.579046] LustreError: 163995:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5256.673289] LustreError: 163996:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5258.350505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5259.356599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5266.799961] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5276.907239] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5283.138347] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5293.122292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5293.405763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5300.471181] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5306.828845] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5311.937050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5322.210644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5322.218886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5322.236198] Lustre: Skipped 1 previous similar message [ 5322.238660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5322.275155] Lustre: Skipped 1 previous similar message [ 5324.685781] Lustre: server umount lustre-OST0000 complete [ 5330.921262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5330.930537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5330.959962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5332.449472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5332.450814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5332.476966] Lustre: Skipped 1 previous similar message [ 5337.574321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5337.576592] Lustre: Skipped 1 previous similar message [ 5342.698108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5342.708852] Lustre: Skipped 1 previous similar message [ 5345.247894] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5345.450895] Lustre: server umount lustre-MDT0000 complete [ 5347.808634] LustreError: 164707:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5347.838972] LustreError: 164707:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5350.382125] LustreError: 163976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778657461 with bad export cookie 7931509988605905940 [ 5350.404188] LustreError: 163976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5350.407886] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5350.994249] Lustre: server umount lustre-MDT0001 complete [ 5363.146307] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5367.122506] Key type lgssc unregistered [ 5367.458848] LNet: 167208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5367.475373] LNetError: 167208:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5367.502726] LNet: Removed LNI 192.168.201.123@tcp [ 5368.389356] Key type .llcrypt unregistered [ 5368.395283] Key type ._llcrypt unregistered [ 5396.971109] Key type ._llcrypt registered [ 5396.976255] Key type .llcrypt registered [ 5397.128306] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5399.168660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5399.200837] alg: No test for adler32 (adler32-zlib) [ 5400.461350] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 5400.960715] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 5402.663176] Key type lgssc registered [ 5404.522568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5418.495697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5418.509643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5420.004752] Lustre: *** cfs_fail_loc=135, val=0*** [ 5420.075953] LustreError: 168716:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5420.088354] LustreError: 168716:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.123@tcp: cfg command failed: rc = -2 [ 5420.114840] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5420.114840] [ 5420.139089] LustreError: MGC192.168.201.123@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5420.153834] LustreError: 168677:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5420.162782] LustreError: 168677:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5420.171939] LustreError: 168677:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 5420.175360] LustreError: 168677:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5420.271658] Lustre: server umount lustre-MDT0000 complete [ 5420.276397] LustreError: 168677:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5428.041242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5428.652798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5433.745585] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5438.956229] LustreError: 169088:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.984939] LustreError: 169088:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5444.068610] LustreError: 169087:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5444.802781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5445.329475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5449.683311] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5456.330520] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5461.649993] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5466.080758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5466.087052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5466.100798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5466.124763] Lustre: Skipped 1 previous similar message [ 5470.244206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5470.251271] Lustre: Skipped 2 previous similar messages [ 5471.030181] Lustre: server umount lustre-MDT0000 complete [ 5475.297801] LustreError: 170091:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5475.323049] LustreError: 170091:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5475.900334] LustreError: 169805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778657586 with bad export cookie 17392145891832305810 [ 5475.911131] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5476.521842] Lustre: server umount lustre-MDT0001 complete [ 5487.766271] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5490.747583] Key type lgssc unregistered [ 5491.182835] LNet: 171168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5491.188451] LNetError: 171168:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5491.204890] LNet: Removed LNI 192.168.201.123@tcp [ 5492.145351] Key type .llcrypt unregistered [ 5492.146883] Key type ._llcrypt unregistered [ 5517.874887] Key type ._llcrypt registered [ 5517.877604] Key type .llcrypt registered [ 5517.997229] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5518.358307] Key type .llcrypt unregistered [ 5518.360594] Key type ._llcrypt unregistered [ 5536.285507] Key type ._llcrypt registered [ 5536.289707] Key type .llcrypt registered [ 5536.412989] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5537.615611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5537.662599] alg: No test for adler32 (adler32-zlib) [ 5538.939705] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 5539.339468] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 5541.151379] Key type lgssc registered [ 5542.638843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5551.340705] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5551.361475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5566.880423] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5571.999220] LustreError: 173269:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 5598.623542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5605.812979] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5613.516683] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5621.775368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5622.273335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5622.368223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5626.709804] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5627.367112] LustreError: 173294:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5627.391275] LustreError: 173294:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5632.482764] LustreError: 173295:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5633.515100] LustreError: 174194:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5633.545021] LustreError: 174194:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5636.321071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.757340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5641.352544] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5648.178991] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5652.669661] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5656.160593] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5667.299857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5667.306304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5667.326586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5667.814606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5667.843059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5671.461451] Lustre: server umount lustre-OST0000 complete [ 5677.543773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5677.562889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5677.593367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5682.039496] Lustre: server umount lustre-MDT0000 complete [ 5683.182378] LustreError: 174197:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5683.199405] LustreError: 174197:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5686.785383] LustreError: 173275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778657797 with bad export cookie 5301941443917170026 [ 5686.795444] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5686.799893] LustreError: 173275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5687.283440] Lustre: server umount lustre-MDT0001 complete [ 5699.928257] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5703.751603] Key type lgssc unregistered [ 5704.169678] LNet: 176829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5704.174612] LNetError: 176829:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5704.195847] LNet: Removed LNI 192.168.201.123@tcp [ 5705.484546] Key type .llcrypt unregistered [ 5705.488156] Key type ._llcrypt unregistered [ 5729.900955] Key type ._llcrypt registered [ 5729.904589] Key type .llcrypt registered [ 5730.015868] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5731.711333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5731.733293] alg: No test for adler32 (adler32-zlib) [ 5732.903465] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 5733.194100] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 5734.847293] Key type lgssc registered [ 5736.129580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5745.318147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5745.369051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5746.938821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5751.422838] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5756.394614] LustreError: 178224:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5756.427115] LustreError: 178224:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5761.247359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5761.514316] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5761.688840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5765.925586] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5772.804460] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5776.626960] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5783.852592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5784.269286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5791.502431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5791.754369] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5798.554731] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5804.995787] Lustre: Failing over lustre-MDT0000 [ 5805.614324] Lustre: server umount lustre-MDT0000 complete [ 5806.354134] LustreError: 179839:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5807.075899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5807.092155] Lustre: Skipped 2 previous similar messages [ 5811.470105] LustreError: 178220:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5811.501270] LustreError: 178220:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5816.588804] LustreError: 180151:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5816.614823] LustreError: 180151:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5821.707347] LustreError: 178221:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5821.734351] LustreError: 178221:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5823.457561] Lustre: 177537:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778657918/real 1778657918] req@00000000a182e28e x1865057927455616/t0(0) o400->MGC192.168.201.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1778657934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5823.501887] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5827.749621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5831.940796] LustreError: 179839:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5831.971439] LustreError: 179839:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 12 previous similar messages [ 5833.711237] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa4ffeecb9f345540 [ 5833.724952] Lustre: MGC192.168.201.123@tcp: Connection restored to 0@lo (at 0@lo) [ 5834.045565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5834.129662] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5837.054136] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5838.495569] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5839.341411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5839.366511] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5839.423230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5843.068826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5851.047314] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5863.906531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5863.921824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5863.946144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5864.937367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5864.957201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5864.962644] Lustre: Skipped 1 previous similar message [ 5869.925311] Lustre: server umount lustre-OST0000 complete [ 5875.169073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5875.170222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5875.179090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5875.192154] Lustre: Skipped 1 previous similar message [ 5875.220338] Lustre: Skipped 1 previous similar message [ 5880.289657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5880.307841] Lustre: Skipped 1 previous similar message [ 5880.935256] Lustre: server umount lustre-MDT0000 complete [ 5885.413171] LustreError: 179839:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5885.443447] LustreError: 179839:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 5885.488578] LustreError: 181448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778657996 with bad export cookie 11889484099598177600 [ 5885.493281] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5885.505511] LustreError: 181448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5886.131761] Lustre: server umount lustre-MDT0001 complete [ 5899.624167] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 5902.560116] Key type lgssc unregistered [ 5902.790780] LNet: 182761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5902.795696] LNetError: 182761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5902.807780] LNet: Removed LNI 192.168.201.123@tcp [ 5903.661767] Key type .llcrypt unregistered [ 5903.666375] Key type ._llcrypt unregistered [ 5931.654384] Key type ._llcrypt registered [ 5931.657990] Key type .llcrypt registered [ 5931.771780] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 5932.968863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5932.990736] alg: No test for adler32 (adler32-zlib) [ 5934.000559] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 5934.200245] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 5935.879363] Key type lgssc registered [ 5937.096870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5944.638941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5944.653754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.078521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.365448] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5956.579438] LustreError: 184160:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.592096] LustreError: 184160:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5961.705469] LustreError: 184159:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5962.177156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5962.677325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5967.902993] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5973.820964] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5977.828829] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5984.845945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5985.184286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5991.444886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5991.872811] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 5998.136963] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6002.783935] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6010.012660] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6020.916098] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6032.356279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6032.357298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6032.358572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6032.410593] Lustre: Skipped 1 previous similar message [ 6036.767169] Lustre: server umount lustre-OST0000 complete [ 6042.596891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6042.605759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6042.608353] Lustre: Skipped 1 previous similar message [ 6042.619311] Lustre: Skipped 2 previous similar messages [ 6047.082810] Lustre: server umount lustre-MDT0000 complete [ 6047.720594] LustreError: 185913:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6047.741882] LustreError: 185913:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6051.730543] LustreError: 184139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778658162 with bad export cookie 9858249243268729565 [ 6051.739744] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6051.751578] LustreError: 184139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6052.813326] Lustre: server umount lustre-MDT0001 complete [ 6069.295941] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 6073.150271] Key type lgssc unregistered [ 6073.383840] LNet: 187511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6073.388771] LNetError: 187511:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6073.403851] LNet: Removed LNI 192.168.201.123@tcp [ 6074.087330] Key type .llcrypt unregistered [ 6074.096180] Key type ._llcrypt unregistered [ 6101.620182] Key type ._llcrypt registered [ 6101.622802] Key type .llcrypt registered [ 6101.749393] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 6103.059422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6103.293306] alg: No test for adler32 (adler32-zlib) [ 6104.295653] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 6104.475774] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 6106.135235] Key type lgssc registered [ 6107.411852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6118.831080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6118.872398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6120.832373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6126.778388] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6130.155526] LustreError: 188910:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6130.186504] LustreError: 188910:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6135.264319] LustreError: 188911:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6137.528405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6138.120278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6142.970643] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6149.144516] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6153.183346] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6160.886145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6161.446688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6165.574599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6169.286627] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6177.329104] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6182.898823] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6196.217522] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6209.478559] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6215.682611] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6226.920363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.938641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6226.942014] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6232.034036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6232.056891] Lustre: Skipped 2 previous similar messages [ 6232.817112] Lustre: server umount lustre-OST0000 complete [ 6240.737838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6240.741789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6240.756481] Lustre: Skipped 1 previous similar message [ 6240.762667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6242.275125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6243.918293] Lustre: server umount lustre-MDT0000 complete [ 6247.399856] LustreError: 188911:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6247.430503] LustreError: 188911:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6248.882507] LustreError: 188891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778658359 with bad export cookie 10153287251841652019 [ 6248.889132] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6248.899488] LustreError: 188891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6249.427984] Lustre: server umount lustre-MDT0001 complete [ 6261.789340] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 6264.613105] Key type lgssc unregistered [ 6264.911790] LNet: 193526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6264.917146] LNetError: 193526:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6264.947955] LNet: Removed LNI 192.168.201.123@tcp [ 6266.236333] Key type .llcrypt unregistered [ 6266.241090] Key type ._llcrypt unregistered [ 6295.573714] Key type ._llcrypt registered [ 6295.580067] Key type .llcrypt registered [ 6295.796123] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 6297.220604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6297.248227] alg: No test for adler32 (adler32-zlib) [ 6298.395165] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 6298.664967] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 6300.439267] Key type lgssc registered [ 6301.842279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6309.377379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6309.404412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6311.113738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6315.053583] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6321.638027] LustreError: 194924:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6321.673629] LustreError: 194924:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6323.747220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6324.048222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6329.309133] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6335.859101] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6339.972099] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6347.645503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6348.243930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6356.483213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6357.956666] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6364.840804] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6385.145238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6385.163924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6385.203068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6387.180697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6387.182421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.183871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6388.878622] Lustre: server umount lustre-OST0000 complete [ 6395.872287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6395.879990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6395.898365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6395.910937] Lustre: Skipped 1 previous similar message [ 6402.530530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6402.538018] Lustre: Skipped 3 previous similar messages [ 6407.136143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6407.502816] Lustre: server umount lustre-MDT0000 complete [ 6407.656921] LustreError: 194925:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6407.696092] LustreError: 194925:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6412.753478] LustreError: 194903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778658523 with bad export cookie 1693961890625128312 [ 6412.766507] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6412.769138] LustreError: 194903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6413.567977] Lustre: server umount lustre-MDT0001 complete [ 6427.362189] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 6430.627886] Key type lgssc unregistered [ 6430.989650] LNet: 198254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6431.005286] LNetError: 198254:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6431.026270] LNet: Removed LNI 192.168.201.123@tcp [ 6432.116122] Key type .llcrypt unregistered [ 6432.118150] Key type ._llcrypt unregistered [ 6452.126524] Key type ._llcrypt registered [ 6452.129362] Key type .llcrypt registered [ 6452.234266] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 6453.414137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6453.444194] alg: No test for adler32 (adler32-zlib) [ 6454.496216] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 6454.705434] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 6456.367157] Key type lgssc registered [ 6457.364402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6465.037478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6465.054852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6466.634993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6471.171975] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6476.772805] LustreError: 199505:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6476.787750] LustreError: 199505:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6477.794988] LustreError: 199504:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6479.494702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6479.798808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6484.311735] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6491.619412] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6495.544165] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6502.200733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6502.624867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6508.369321] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6512.892155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6513.939943] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6530.528948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6530.538618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6530.562396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6533.602063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6533.604673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6533.618518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6536.270187] Lustre: server umount lustre-OST0000 complete [ 6541.281054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6541.295117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6541.317462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6541.330354] Lustre: Skipped 1 previous similar message [ 6543.841079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6543.842747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6543.864824] Lustre: Skipped 1 previous similar message [ 6546.820729] Lustre: server umount lustre-MDT0000 complete [ 6548.966187] LustreError: 199505:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6548.977449] LustreError: 199505:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6551.053953] LustreError: 199485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778658662 with bad export cookie 11912684468613118704 [ 6551.076018] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6551.604671] Lustre: server umount lustre-MDT0001 complete [ 6563.373810] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 6567.197702] Key type lgssc unregistered [ 6567.630984] LNet: 202833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6567.652585] LNetError: 202833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6567.676890] LNet: Removed LNI 192.168.201.123@tcp [ 6568.628859] Key type .llcrypt unregistered [ 6568.631208] Key type ._llcrypt unregistered [ 6588.424399] Key type ._llcrypt registered [ 6588.426766] Key type .llcrypt registered [ 6588.520995] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 6589.795860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6589.942266] alg: No test for adler32 (adler32-zlib) [ 6591.059080] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 6591.391481] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 6593.144126] Key type lgssc registered [ 6594.332953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6603.347166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6603.370810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6605.020893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6609.745091] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6615.529324] LustreError: 204082:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6615.546868] LustreError: 204082:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6618.696446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.046536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6624.369850] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6630.640631] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6635.127629] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6641.735586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6642.114197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6647.134397] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6650.309689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6652.170351] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6661.634853] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.712154] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6661.901163] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6667.975235] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6674.734554] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6676.460334] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6688.844686] Lustre: Permanently deactivating lustre-OST0001 [ 6688.880234] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6696.012601] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6705.215177] Lustre: Permanently reactivating lustre-OST0001 [ 6705.232231] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6705.239991] Lustre: Skipped 2 previous similar messages [ 6711.428741] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6711.451483] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6711.455726] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6711.480943] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6711.593437] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6721.007753] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6721.031944] Lustre: Skipped 1 previous similar message [ 6721.042208] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6721.053756] Lustre: Skipped 1 previous similar message [ 6724.718136] Lustre: server umount lustre-OST0001 complete [ 6726.113438] LustreError: 205679:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6726.137063] LustreError: 205679:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6731.234194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6731.245312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6731.248910] LustreError: 206719:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6731.264519] Lustre: Skipped 1 previous similar message [ 6731.271975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6731.300413] Lustre: Skipped 1 previous similar message [ 6735.838775] Lustre: server umount lustre-OST0000 complete [ 6741.479653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6741.483819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6741.504110] Lustre: Skipped 1 previous similar message [ 6745.886553] Lustre: server umount lustre-MDT0000 complete [ 6746.592163] LustreError: 204082:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6746.628851] LustreError: 204082:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6750.482908] LustreError: 204062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778658861 with bad export cookie 13811565022628331702 [ 6750.495757] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6750.498962] LustreError: 204062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6750.891294] Lustre: server umount lustre-MDT0001 complete [ 6761.813723] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 6764.638373] Key type lgssc unregistered [ 6764.978340] LNet: 208681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6764.983619] LNetError: 208681:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6764.993400] LNet: Removed LNI 192.168.201.123@tcp [ 6765.917833] Key type .llcrypt unregistered [ 6765.923090] Key type ._llcrypt unregistered [ 6805.147232] Key type ._llcrypt registered [ 6805.150310] Key type .llcrypt registered [ 6805.339787] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 6807.517657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6807.655636] alg: No test for adler32 (adler32-zlib) [ 6808.648431] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 6808.854711] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 6810.535187] Key type lgssc registered [ 6811.729862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6819.111554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6819.137915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6820.388513] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6820.415826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6820.592307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6825.170950] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6835.753611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6835.952816] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6835.995317] Lustre: 210883:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6836.372892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6841.038201] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6846.793731] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6850.214187] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6856.091822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6856.335134] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6856.577096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6862.263292] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 6862.978267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6867.189029] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6869.923311] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6876.726304] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6882.346541] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6891.250776] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6901.307377] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6919.856994] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6919.869804] Lustre: Skipped 1 previous similar message [ 6955.801139] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6955.805272] Lustre: Skipped 3 previous similar messages [ 7021.193891] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7021.200622] Lustre: Skipped 8 previous similar messages [ 7052.258985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7052.271971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.281927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7052.781952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7052.783669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.818379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7057.334634] Lustre: server umount lustre-OST0000 complete [ 7063.009579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7063.043236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7068.132327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7068.143334] Lustre: Skipped 2 previous similar messages [ 7068.902417] Lustre: server umount lustre-MDT0000 complete [ 7072.728276] LustreError: 212517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778659183 with bad export cookie 16942110389265672562 [ 7072.729598] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7072.741021] LustreError: 212517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7073.030288] Lustre: server umount lustre-MDT0001 complete [ 7083.872399] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 7087.283930] Key type lgssc unregistered [ 7087.667742] LNet: 215334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7087.675300] LNetError: 215334:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7087.689261] LNet: Removed LNI 192.168.201.123@tcp [ 7088.665279] Key type .llcrypt unregistered [ 7088.667548] Key type ._llcrypt unregistered [ 7115.801937] Key type ._llcrypt registered [ 7115.806734] Key type .llcrypt registered [ 7115.947430] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 7117.660219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7117.786669] alg: No test for adler32 (adler32-zlib) [ 7119.039446] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 7119.259880] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 7121.047158] Key type lgssc registered [ 7123.039886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7133.429870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7133.453776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7135.251635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7141.878529] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7145.443601] LustreError: 216732:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7145.470491] LustreError: 216732:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7150.562660] LustreError: 216731:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7153.456825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7153.829851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7158.489967] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7164.979278] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7168.822545] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7177.750945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7178.211353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7180.280741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7184.581327] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7190.060831] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7208.802965] Lustre: server umount lustre-OST0000 complete [ 7215.072923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7215.084091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7215.125839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7216.105691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7216.111834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7216.133436] Lustre: Skipped 1 previous similar message [ 7221.218683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7221.221594] Lustre: Skipped 1 previous similar message [ 7222.343368] Lustre: server umount lustre-MDT0000 complete [ 7226.335699] Lustre: 216042:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778659321/real 1778659321] req@00000000cf7c97c7 x1865059380793344/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1778659337 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7226.350827] LustreError: 217723:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7226.387903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7226.433810] LustreError: 217723:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7227.024117] LustreError: 218981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778659338 with bad export cookie 13890119434087115765 [ 7227.041598] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7227.043082] LustreError: 218981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7227.504278] Lustre: server umount lustre-MDT0001 complete [ 7239.491484] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 7243.206628] Key type lgssc unregistered [ 7243.692866] LNet: 220073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7243.698834] LNetError: 220073:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7243.720524] LNet: Removed LNI 192.168.201.123@tcp [ 7244.906835] Key type .llcrypt unregistered [ 7244.909504] Key type ._llcrypt unregistered [ 7268.362931] Key type ._llcrypt registered [ 7268.368399] Key type .llcrypt registered [ 7268.486348] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 7269.974153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7270.075272] alg: No test for adler32 (adler32-zlib) [ 7271.076665] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 7271.272596] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 7272.919272] Key type lgssc registered [ 7274.253942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7546.221710] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 7550.159868] Key type lgssc unregistered [ 7550.746427] LNet: 222065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7550.755143] LNetError: 222065:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7550.770690] LNet: Removed LNI 192.168.201.123@tcp [ 7551.845571] Key type .llcrypt unregistered [ 7551.849540] Key type ._llcrypt unregistered [ 7583.697139] Key type ._llcrypt registered [ 7583.698501] Key type .llcrypt registered [ 7583.880985] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 7602.566898] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 7603.838463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7603.946514] alg: No test for adler32 (adler32-zlib) [ 7605.072299] Lustre: Lustre: Build Version: 2.17.53_3_gb43cd8a [ 7605.292084] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 7606.975184] Key type lgssc registered [ 7608.254979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7639.221485] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7341 sec ========== 04:09:07 (1778659747) [ 7640.763858] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:09:09 (1778659749) === [ 7644.811395] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:09:13 (1778659753) === [ 7691.268956] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 7694.714380] Key type lgssc unregistered [ 7695.169969] LNet: 227252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7695.177116] LNetError: 227252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7695.207745] LNet: Removed LNI 192.168.201.123@tcp [ 7696.021211] Key type .llcrypt unregistered [ 7696.023379] Key type ._llcrypt unregistered