[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 509959365 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003212] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.007990] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009029] pid_max: default: 32768 minimum: 301 [ 0.010151] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013092] *** VALIDATE selinux *** [ 0.021621] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026379] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027179] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029126] *** VALIDATE tmpfs *** [ 0.031108] *** VALIDATE proc *** [ 0.032252] *** VALIDATE cgroup *** [ 0.033013] *** VALIDATE cgroup2 *** [ 0.034306] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035178] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038003] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042447] debug: unmapping init [mem 0xffffffff93659000-0xffffffff93660fff] [ 0.045164] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046724] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047031] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054326] rcu: Hierarchical SRCU implementation. [ 0.056561] smp: Bringing up secondary CPUs ... [ 0.057739] x86: Booting SMP configuration: [ 0.058035] .... node #0, CPUs: #1 #2 #3 [ 0.061331] smp: Brought up 1 node, 4 CPUs [ 0.063011] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.133111] node 0 deferred pages initialised in 66ms [ 0.136129] devtmpfs: initialized [ 0.137237] x86/mm: Memory block size: 128MB [ 0.140280] gcov: version magic: 0x41383552 [ 0.142330] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145062] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146212] pinctrl core: initialized pinctrl subsystem [ 0.147141] [ 0.147559] ************************************************************* [ 0.148022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149013] ** ** [ 0.150018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151018] ** ** [ 0.152012] ** This means that this kernel is built to expose internal ** [ 0.153014] ** IOMMU data structures, which may compromise security on ** [ 0.154016] ** your system. ** [ 0.155011] ** ** [ 0.156010] ** If you see this message and you are not debugging the ** [ 0.157015] ** kernel, report this immediately to your vendor! ** [ 0.158013] ** ** [ 0.159012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160013] ************************************************************* [ 0.161933] NET: Registered protocol family 16 [ 0.162469] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.163068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.164087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.165590] cpuidle: using governor menu [ 0.167502] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.169390] PCI: Using configuration type 1 for base access [ 0.171097] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.177122] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.179031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.182144] cryptd: max_cpu_qlen set to 1000 [ 0.184285] ACPI: Added _OSI(Module Device) [ 0.186010] ACPI: Added _OSI(Processor Device) [ 0.187000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.187000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.191183] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.197385] ACPI: Interpreter enabled [ 0.198049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.199011] ACPI: Using IOAPIC for interrupt routing [ 0.201110] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.204328] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.212605] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.214040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.216012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.217078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.221018] acpiphp: Slot [2] registered [ 0.221974] acpiphp: Slot [5] registered [ 0.223079] acpiphp: Slot [6] registered [ 0.224120] acpiphp: Slot [7] registered [ 0.225140] acpiphp: Slot [8] registered [ 0.226156] acpiphp: Slot [9] registered [ 0.227000] acpiphp: Slot [10] registered [ 0.227000] acpiphp: Slot [3] registered [ 0.227114] acpiphp: Slot [4] registered [ 0.228000] acpiphp: Slot [11] registered [ 0.229105] acpiphp: Slot [12] registered [ 0.230071] acpiphp: Slot [13] registered [ 0.230948] acpiphp: Slot [14] registered [ 0.232056] acpiphp: Slot [15] registered [ 0.232827] acpiphp: Slot [16] registered [ 0.234092] acpiphp: Slot [17] registered [ 0.235009] acpiphp: Slot [18] registered [ 0.235818] acpiphp: Slot [19] registered [ 0.236045] acpiphp: Slot [20] registered [ 0.236887] acpiphp: Slot [21] registered [ 0.238056] acpiphp: Slot [22] registered [ 0.238919] acpiphp: Slot [23] registered [ 0.240108] acpiphp: Slot [24] registered [ 0.241042] acpiphp: Slot [25] registered [ 0.241919] acpiphp: Slot [26] registered [ 0.243059] acpiphp: Slot [27] registered [ 0.243868] acpiphp: Slot [28] registered [ 0.244067] acpiphp: Slot [29] registered [ 0.244928] acpiphp: Slot [30] registered [ 0.246070] acpiphp: Slot [31] registered [ 0.247047] PCI host bridge to bus 0000:00 [ 0.247992] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.250020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.252018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.253020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.255017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.256022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.258184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.259720] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.262065] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.268522] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.272068] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.273012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.275012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.276013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.278233] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.281861] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.286055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.289848] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.296014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.308024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.314016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.320206] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.325020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.333021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.358018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.369000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.385018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.396027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.425071] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.453000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.460025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.469020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.490021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.502497] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.510017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.521018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.539016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.553330] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.560023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.566017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.578017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.585232] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.593017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.600021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.615024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.624000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.626378] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.629390] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.632317] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.634299] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.638092] iommu: Default domain type: Passthrough [ 0.639346] SCSI subsystem initialized [ 0.640082] ACPI: bus type USB registered [ 0.641031] usbcore: registered new interface driver usbfs [ 0.642058] usbcore: registered new interface driver hub [ 0.643085] usbcore: registered new device driver usb [ 0.644108] pps_core: LinuxPPS API ver. 1 registered [ 0.646010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.647041] PTP clock support registered [ 0.649119] EDAC MC: Ver: 3.0.0 [ 0.650482] PCI: Using ACPI for IRQ routing [ 0.651540] NetLabel: Initializing [ 0.652007] NetLabel: domain hash size = 128 [ 0.652871] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.653083] NetLabel: unlabeled traffic allowed by default [ 0.654074] vgaarb: loaded [ 0.655214] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.656006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.663000] clocksource: Switched to clocksource kvm-clock [ 0.770860] VFS: Disk quotas dquot_6.6.0 [ 0.773060] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.775193] *** VALIDATE ramfs *** [ 0.776248] *** VALIDATE hugetlbfs *** [ 0.777868] pnp: PnP ACPI init [ 0.780405] pnp: PnP ACPI: found 6 devices [ 0.813424] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.816881] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.819222] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.821629] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.824108] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.826781] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.829802] NET: Registered protocol family 2 [ 0.832223] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.837371] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.841168] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.847088] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.850401] TCP: Hash tables configured (established 65536 bind 65536) [ 0.853280] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.856226] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.858645] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.861477] NET: Registered protocol family 1 [ 0.864674] RPC: Registered named UNIX socket transport module. [ 0.867470] RPC: Registered udp transport module. [ 0.869617] RPC: Registered tcp transport module. [ 0.871971] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.874353] NET: Registered protocol family 44 [ 0.875702] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.877678] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.879434] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.881175] PCI: CLS 0 bytes, default 64 [ 0.882668] Unpacking initramfs... [ 2.246867] debug: unmapping init [mem 0xffff8b71fcc54000-0xffff8b71fffbffff] [ 2.250721] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.252843] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.255785] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.764794] Initialise system trusted keyrings [ 2.766342] Key type blacklist registered [ 2.768067] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.776425] zbud: loaded [ 2.779385] *** VALIDATE nfs *** [ 2.780280] *** VALIDATE nfs4 *** [ 2.781682] pstore: using deflate compression [ 2.784668] Platform Keyring initialized [ 2.894446] NET: Registered protocol family 38 [ 2.896287] Key type asymmetric registered [ 2.897781] Asymmetric key parser 'x509' registered [ 2.899668] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.902546] io scheduler mq-deadline registered [ 2.904329] io scheduler kyber registered [ 2.905770] io scheduler bfq registered [ 2.907237] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.909665] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.912177] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.915061] ACPI: Power Button [PWRF] [ 2.919643] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.926902] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.942709] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.948973] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.972197] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.000192] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.028714] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.033224] Non-volatile memory driver v1.3 [ 3.036434] Linux agpgart interface v0.103 [ 3.064120] virtio_blk virtio1: [vda] 150064 512-byte logical blocks (76.8 MB/73.3 MiB) [ 3.066022] vda: detected capacity change from 0 to 76832768 [ 3.077564] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.080380] vdb: detected capacity change from 0 to 1073741824 [ 3.094254] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.096865] vdc: detected capacity change from 0 to 2621440000 [ 3.109644] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.111377] vdd: detected capacity change from 0 to 2621440000 [ 3.125358] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.127878] vde: detected capacity change from 0 to 4294967296 [ 3.139564] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.141476] vdf: detected capacity change from 0 to 4294967296 [ 3.146487] libphy: Fixed MDIO Bus: probed [ 3.161366] usbcore: registered new interface driver usbserial_generic [ 3.164295] usbserial: USB Serial support registered for generic [ 3.166994] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.172319] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.174625] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.177667] mousedev: PS/2 mouse device common for all mice [ 3.180520] rtc_cmos 00:05: RTC can wake from S4 [ 3.185061] rtc_cmos 00:05: registered as rtc0 [ 3.186084] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.187815] intel_pstate: CPU model not supported [ 3.187898] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.192316] hid: raw HID events driver (C) Jiri Kosina [ 3.194029] usbcore: registered new interface driver usbhid [ 3.194118] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.195295] usbhid: USB HID core driver [ 3.195433] drop_monitor: Initializing network drop monitor service [ 3.200151] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.201508] Initializing XFRM netlink socket [ 3.205333] NET: Registered protocol family 10 [ 3.208100] Segment Routing with IPv6 [ 3.208966] NET: Registered protocol family 17 [ 3.210188] mpls_gso: MPLS GSO support [ 3.214514] RAS: Correctable Errors collector initialized. [ 3.215945] AVX version of gcm_enc/dec engaged. [ 3.217026] AES CTR mode by8 optimization enabled [ 3.284190] sched_clock: Marking stable (3284166042, 0)->(4212817168, -928651126) [ 3.286738] registered taskstats version 1 [ 3.289541] Loading compiled-in X.509 certificates [ 3.291770] zswap: loaded using pool lzo/zbud [ 3.312567] Key type big_key registered [ 3.322969] Key type encrypted registered [ 3.324041] ima: No TPM chip found, activating TPM-bypass! [ 3.325512] ima: Allocated hash algorithm: sha1 [ 3.326563] ima: No architecture policies found [ 3.327792] evm: Initialising EVM extended attributes: [ 3.329104] evm: security.selinux [ 3.329835] evm: security.ima [ 3.330451] evm: security.capability [ 3.331195] evm: HMAC attrs: 0x1 [ 3.333140] rtc_cmos 00:05: setting system clock to 2026-09-07 02:27:19 UTC (1788748039) [ 3.339267] debug: unmapping init [mem 0xffffffff94603000-0xffffffff947fffff] [ 3.342401] debug: unmapping init [mem 0xffffffff93382000-0xffffffff93658fff] [ 3.351197] Write protecting the kernel read-only data: 28672k [ 3.354778] debug: unmapping init [mem 0xffffffff91a03000-0xffffffff91bfffff] [ 3.357748] debug: unmapping init [mem 0xffffffff92314000-0xffffffff923fffff] [ 3.394334] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.403385] systemd[1]: Detected virtualization kvm. [ 3.405412] systemd[1]: Detected architecture x86-64. [ 3.407553] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.433100] systemd[1]: No hostname configured. [ 3.434479] systemd[1]: Set hostname to . [ 3.435876] random: systemd: uninitialized urandom read (16 bytes read) [ 3.437441] systemd[1]: Initializing machine ID from random generator. [ 3.484961] random: ln: uninitialized urandom read (6 bytes read) [ 3.571398] random: systemd: uninitialized urandom read (16 bytes read) [ 3.573633] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.580750] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.585216] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.168305] device-mapper: uevent: version 1.0.3 [ 4.170412] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.791551] virtio_net virtio0 ens2: renamed from eth0 [ 4.800418] random: fast init done [ 4.836752] scsi host0: ata_piix [ 4.860252] scsi host1: ata_piix [ 4.861947] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.864506] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.092663] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.853183] random: crng init done [ 9.857206] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.638640] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.508970] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.523937] SELinux: Disabled at runtime. [ 16.647269] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.666143] systemd[1]: Detected virtualization kvm. [ 16.672190] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.748989] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.760702] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.771630] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.784240] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.791191] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.816535] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.834656] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ 19.244684] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 20.412350] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.489449] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.503908] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.896093] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.955497] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 27.331562] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 28.356548] NFS: Registering the id_resolver key type [ 28.359330] Key type id_resolver registered [ 28.362289] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg126-server login: [ 85.754735] libcfs: loading out-of-tree module taints kernel. [ 85.813880] Key type ._llcrypt registered [ 85.815900] Key type .llcrypt registered [ 85.905320] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 109.818715] hrtimer: interrupt took 11583575 ns [ 112.536702] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 114.342875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.354211] alg: No test for adler32 (adler32-zlib) [ 116.261026] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 117.021283] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 118.831270] Key type lgssc registered [ 121.206820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 138.012870] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 177.161272] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 192.289610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 192.316766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.606933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.631163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.718506] Lustre: lustre-MDT0000: new disk, initializing [ 193.872469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.919060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 199.095248] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.457207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.718642] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 217.769553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 217.785088] Lustre: Skipped 1 previous similar message [ 217.872438] Lustre: lustre-MDT0001: new disk, initializing [ 218.024202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 218.082120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 218.107195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 224.434586] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.501961] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 239.159627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.404806] Lustre: lustre-OST0000: new disk, initializing [ 239.406853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 239.409733] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.452950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 247.791991] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.939054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 248.964546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 249.068644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 263.781914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 264.071907] Lustre: lustre-OST0001: new disk, initializing [ 264.083337] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 264.097737] Lustre: 9518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 264.287599] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 269.901237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 269.910227] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 270.058379] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 272.264535] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 288.384314] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 297.461632] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 304.937807] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing check_logdir /tmp/testlogs/ [ 310.852098] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing yml_node [ 315.818812] Lustre: DEBUG MARKER: Client: 2.17.58.41 [ 319.220898] Lustre: DEBUG MARKER: MDS: 2.17.58.41 [ 322.353147] Lustre: DEBUG MARKER: OSS: 2.17.58.41 [ 324.503255] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 22:32:39 EDT 2026 [ 344.154350] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 346.348344] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 366.559749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 366.566845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.588586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.596816] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.613086] Lustre: Skipped 1 previous similar message [ 367.613105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.644332] Lustre: Skipped 3 previous similar messages [ 370.748201] Lustre: server umount lustre-MDT0000 complete [ 375.208849] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748411 with bad export cookie 6647743167318307778 [ 375.220059] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 375.221789] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 377.824044] LustreError: 6513:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 377.824630] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.837656] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 377.840862] LustreError: 6513:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 377.870444] Lustre: Skipped 1 previous similar message [ 380.831888] LustreError: 6512:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.832094] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 380.851926] LustreError: 6512:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 380.878750] Lustre: Skipped 2 previous similar messages [ 381.644416] Lustre: server umount lustre-MDT0001 complete [ 385.020699] Lustre: server umount lustre-OST0000 complete [ 388.677734] Lustre: server umount lustre-OST0001 complete [ 395.460439] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 405.095663] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 443.478632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.752477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 443.793699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 443.905858] Lustre: lustre-MDT0000: new disk, initializing [ 443.988986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.010881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 449.687336] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 461.942743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.078945] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 462.150194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 462.155137] Lustre: Skipped 1 previous similar message [ 462.290344] Lustre: lustre-MDT0001: new disk, initializing [ 462.423333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 462.481862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 462.494286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 467.592944] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 477.530555] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 483.563988] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 490.770967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 490.982876] Lustre: lustre-OST0000: new disk, initializing [ 490.989532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 490.995172] Lustre: 17040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 491.091432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.959372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 492.972327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 493.073611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 497.392778] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 507.267711] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 513.023086] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 513.289629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 518.585596] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 518.789821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 523.749081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 523.755473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.767599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 527.566806] Lustre: server umount lustre-OST0000 complete [ 533.984553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.996646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 533.997314] Lustre: Skipped 2 previous similar messages [ 533.998407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 534.004481] Lustre: Skipped 2 previous similar messages [ 534.020173] LustreError: Skipped 1 previous similar message [ 546.271318] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 546.481459] Lustre: server umount lustre-MDT0000 complete [ 549.357812] LustreError: 15083:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.381953] LustreError: 15083:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 550.364806] LustreError: 17039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748586 with bad export cookie 6647743167318309311 [ 550.371200] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.372615] LustreError: 17039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 550.936683] Lustre: server umount lustre-MDT0001 complete [ 554.387575] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:36:29 (1788748589) [ 561.340247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.706031] LustreError: 19307:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.816513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.015864] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 567.269351] LustreError: 19308:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.481813] LustreError: 19307:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.516656] LustreError: 19307:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 576.550811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.215373] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 590.742739] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 597.952049] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 605.316885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 612.444895] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 620.707643] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 632.290600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.293575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 632.336150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 632.353922] Lustre: Skipped 5 previous similar messages [ 637.907162] Lustre: server umount lustre-OST0000 complete [ 648.721260] Lustre: server umount lustre-MDT0000 complete [ 652.769182] LustreError: 20278:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.786317] LustreError: 20278:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 653.683840] LustreError: 19288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748689 with bad export cookie 6647743167318310193 [ 653.706050] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 654.326485] Lustre: server umount lustre-MDT0001 complete [ 667.313923] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 670.782510] Key type lgssc unregistered [ 671.146611] LNet: 22843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.154701] LNetError: 22843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.165854] LNet: Removed LNI 192.168.201.126@tcp [ 672.426176] Key type .llcrypt unregistered [ 672.432397] Key type ._llcrypt unregistered [ 698.128930] Key type ._llcrypt registered [ 698.139388] Key type .llcrypt registered [ 698.265121] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 699.735269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 699.902679] alg: No test for adler32 (adler32-zlib) [ 700.974824] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 701.166383] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 702.871199] Key type lgssc registered [ 704.359978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 713.568924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 713.583518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.087434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.005462] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 725.480725] LustreError: 24258:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 725.497249] LustreError: 24258:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 729.235594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 729.798691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 735.388335] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 743.612682] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 749.666543] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 755.665191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 756.066860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.204485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 762.130763] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 768.921112] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 784.865470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 784.867194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 784.889683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 789.596644] Lustre: server umount lustre-OST0000 complete [ 795.120598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 795.121393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 795.137397] Lustre: Skipped 2 previous similar messages [ 795.163722] Lustre: Skipped 2 previous similar messages [ 800.224793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 800.237358] Lustre: Skipped 1 previous similar message [ 805.346905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 805.357627] Lustre: Skipped 1 previous similar message [ 807.391922] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 807.546412] Lustre: server umount lustre-MDT0000 complete [ 810.464641] LustreError: 24985:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 810.481060] LustreError: 24985:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 810.944897] LustreError: 24237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748847 with bad export cookie 15996446244308495422 [ 810.952284] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 810.958244] LustreError: 24237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 811.249906] Lustre: server umount lustre-MDT0001 complete [ 821.833138] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 824.580582] Key type lgssc unregistered [ 824.863623] LNet: 28231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.873924] LNetError: 28231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.893402] LNet: Removed LNI 192.168.201.126@tcp [ 826.621454] Key type .llcrypt unregistered [ 826.623442] Key type ._llcrypt unregistered [ 846.292188] Key type ._llcrypt registered [ 846.294868] Key type .llcrypt registered [ 846.385525] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 847.156745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 847.308797] alg: No test for adler32 (adler32-zlib) [ 848.499221] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 848.728381] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 850.415777] Key type lgssc registered [ 851.360916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 858.201405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 858.215072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 859.637198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.842987] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 869.856096] LustreError: 29618:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 869.884192] LustreError: 29618:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 871.666322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.951070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 875.779901] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 882.395369] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 887.644280] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 899.587459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 899.919582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.068500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 905.180588] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 911.441400] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 925.156736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 925.167770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.177265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 927.711963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 927.716580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 927.717115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 927.721159] Lustre: Skipped 1 previous similar message [ 929.727407] Lustre: server umount lustre-OST0000 complete [ 932.842780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.846650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 932.849576] Lustre: Skipped 1 previous similar message [ 932.866790] Lustre: Skipped 1 previous similar message [ 937.952963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 937.959269] Lustre: Skipped 1 previous similar message [ 943.072846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 947.169762] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 947.307396] Lustre: server umount lustre-MDT0000 complete [ 948.198893] LustreError: 29619:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.218891] LustreError: 29619:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 950.476784] LustreError: 29599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748986 with bad export cookie 3977172669223023899 [ 950.486259] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 950.487593] LustreError: 29599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 950.862744] Lustre: server umount lustre-MDT0001 complete [ 958.457836] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 960.921891] Key type lgssc unregistered [ 961.159919] LNet: 33509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.167855] LNetError: 33509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.183577] LNet: Removed LNI 192.168.201.126@tcp [ 962.110486] Key type .llcrypt unregistered [ 962.112487] Key type ._llcrypt unregistered [ 980.627529] Key type ._llcrypt registered [ 980.629220] Key type .llcrypt registered [ 980.744795] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 981.861798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 981.885066] alg: No test for adler32 (adler32-zlib) [ 982.902265] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 983.093831] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 984.759391] Key type lgssc registered [ 985.642192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 991.728983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 991.764579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 993.271939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 996.967193] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1002.764980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.003894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1006.754691] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1012.448199] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1016.835902] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1022.775277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1023.031430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1027.674117] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1034.391250] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1039.420875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1048.543955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1048.548302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1048.560113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1049.570907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1049.571537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1049.573800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.604658] Lustre: Skipped 1 previous similar message [ 1054.689037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1054.696129] Lustre: Skipped 1 previous similar message [ 1059.295247] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1059.412888] Lustre: server umount lustre-OST0000 complete [ 1064.426218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1064.443173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1064.456157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.593134] Lustre: server umount lustre-MDT0000 complete [ 1070.048444] LustreError: 36430:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1070.071370] LustreError: 36430:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 1071.026688] LustreError: 34879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749107 with bad export cookie 6578795134478774758 [ 1071.033207] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1071.354173] Lustre: server umount lustre-MDT0001 complete [ 1079.211831] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1081.759539] Key type lgssc unregistered [ 1081.989891] LNet: 38430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.997312] LNetError: 38430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1082.010892] LNet: Removed LNI 192.168.201.126@tcp [ 1082.678145] Key type .llcrypt unregistered [ 1082.680326] Key type ._llcrypt unregistered [ 1099.359872] Key type ._llcrypt registered [ 1099.362252] Key type .llcrypt registered [ 1099.437685] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1100.420140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1100.451792] alg: No test for adler32 (adler32-zlib) [ 1101.448373] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1101.652160] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1103.312122] Key type lgssc registered [ 1104.331386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1110.824385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1110.840245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.214429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.213739] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1122.795432] LustreError: 39818:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1122.809628] LustreError: 39818:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1123.246654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.490709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1127.448788] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1132.740205] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1136.729731] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1141.571604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1141.837087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.064713] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1151.535736] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1157.241185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1167.438734] Lustre: lustre-OST0000: Not available for connect from 192.168.201.26@tcp (stopping) [ 1167.849505] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1167.865739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1172.524293] Lustre: lustre-OST0000: Not available for connect from 192.168.201.26@tcp (stopping) [ 1172.527097] Lustre: Skipped 2 previous similar messages [ 1172.631461] Lustre: server umount lustre-OST0000 complete [ 1190.369257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1190.377127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.385761] Lustre: Skipped 1 previous similar message [ 1190.389468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.447201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.452612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.560210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1198.567247] Lustre: Skipped 2 previous similar messages [ 1203.167173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1203.320177] Lustre: server umount lustre-MDT0000 complete [ 1203.680750] LustreError: 39814:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.692824] LustreError: 39814:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1205.890291] LustreError: 39797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749242 with bad export cookie 369119760091390266 [ 1205.895282] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1212.167803] Lustre: server umount lustre-MDT0001 complete [ 1220.840229] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1222.870260] Key type lgssc unregistered [ 1223.067479] LNet: 43544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.090437] LNetError: 43544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1224.101607] LNet: Removed LNI 192.168.201.126@tcp [ 1224.780517] Key type .llcrypt unregistered [ 1224.782197] Key type ._llcrypt unregistered [ 1242.033871] Key type ._llcrypt registered [ 1242.036181] Key type .llcrypt registered [ 1242.106623] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1242.786281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1242.847995] alg: No test for adler32 (adler32-zlib) [ 1243.833612] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1244.009286] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1245.679289] Key type lgssc registered [ 1246.585770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1251.327285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1251.339043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1252.607824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.457585] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1261.087117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1261.369486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1264.707338] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1270.118882] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1273.979369] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1278.714510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1278.953479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1283.135145] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1288.367492] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1297.406175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1305.151266] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.26@tcp (stopping) [ 1307.617262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.617682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1307.619870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1307.624826] Lustre: Skipped 1 previous similar message [ 1309.744703] Lustre: server umount lustre-MDT0000 complete [ 1310.254319] LustreError: 44928:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1310.270603] LustreError: 44928:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1312.074249] LustreError: 47531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749348 with bad export cookie 11639346680726529015 [ 1312.075951] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1312.081027] LustreError: 47531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1312.738805] LustreError: 44929:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1312.742301] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1312.766452] Lustre: Skipped 1 previous similar message [ 1312.777556] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1312.785292] Lustre: Skipped 2 previous similar messages [ 1315.368586] LustreError: 44929:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1315.408846] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.26@tcp (stopping) [ 1317.857076] LustreError: 44930:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1318.313512] Lustre: server umount lustre-MDT0001 complete [ 1338.229467] Lustre: server umount lustre-OST0000 complete [ 1346.754995] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1348.181634] Key type lgssc unregistered [ 1348.378525] LNet: 48809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.383752] LNetError: 48809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1348.402845] LNet: Removed LNI 192.168.201.126@tcp [ 1348.933247] Key type .llcrypt unregistered [ 1348.934992] Key type ._llcrypt unregistered [ 1363.013995] Key type ._llcrypt registered [ 1363.016480] Key type .llcrypt registered [ 1363.081750] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1363.613217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1363.624241] alg: No test for adler32 (adler32-zlib) [ 1364.518407] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1364.646792] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1366.279209] Key type lgssc registered [ 1366.960060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1371.223176] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1371.236912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1386.719391] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1391.839671] LustreError: 50171:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 1418.527267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.909612] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1426.682436] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1592.241380] Lustre: server umount lustre-OST0000 complete [ 1613.661285] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1617.140526] Key type lgssc unregistered [ 1617.710390] LNet: 51843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.723189] LNetError: 51843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.764191] LNet: Removed LNI 192.168.201.126@tcp [ 1618.778901] Key type .llcrypt unregistered [ 1618.785804] Key type ._llcrypt unregistered [ 1644.473771] Key type ._llcrypt registered [ 1644.477303] Key type .llcrypt registered [ 1644.557699] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1645.719288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1645.768574] alg: No test for adler32 (adler32-zlib) [ 1646.802503] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1647.042683] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1648.737796] Key type lgssc registered [ 1649.884600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1658.017730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1658.030590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1659.562191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.565191] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1664.993328] LustreError: 53253:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1665.005457] LustreError: 53253:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1669.111718] LustreError: 53254:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1673.372859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1673.915493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1679.067859] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1687.305550] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1693.123209] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1700.360254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1700.745509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1704.824843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1707.681679] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1715.404322] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1718.499440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1718.500885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1718.533571] Lustre: Skipped 1 previous similar message [ 1723.729468] Lustre: server umount lustre-OST0000 complete [ 1728.993137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.004929] Lustre: Skipped 1 previous similar message [ 1729.009749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.119415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.128291] Lustre: Skipped 2 previous similar messages [ 1734.445161] Lustre: server umount lustre-MDT0000 complete [ 1738.219076] LustreError: 56031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749774 with bad export cookie 5212174891461730515 [ 1738.224358] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1738.232880] LustreError: 56031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1738.511923] Lustre: server umount lustre-MDT0001 complete [ 1749.867892] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1752.720478] Key type lgssc unregistered [ 1753.075342] LNet: 56781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.080256] LNetError: 56781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.155724] LNet: Removed LNI 192.168.201.126@tcp [ 1755.270573] Key type .llcrypt unregistered [ 1755.273714] Key type ._llcrypt unregistered [ 1779.222521] Key type ._llcrypt registered [ 1779.228658] Key type .llcrypt registered [ 1779.309070] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1780.633686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1780.732555] alg: No test for adler32 (adler32-zlib) [ 1781.960741] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1782.133580] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1783.815290] Key type lgssc registered [ 1785.119813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1793.270581] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1793.295445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1808.863593] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1813.983362] LustreError: 58163:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 1840.671214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.635885] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1858.090117] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1865.237474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.706665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.772627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1870.825467] LustreError: 58189:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1870.834974] LustreError: 58189:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1870.956151] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1874.925610] LustreError: 58188:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1874.963625] LustreError: 58188:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1879.231608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.530874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1884.453996] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1891.348863] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1896.424934] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1900.017269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.020947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1900.032864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1900.041189] Lustre: Skipped 1 previous similar message [ 1904.477953] Lustre: server umount lustre-OST0000 complete [ 1930.720954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1930.724756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.734813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1930.755521] Lustre: Skipped 2 previous similar messages [ 1930.771342] Lustre: Skipped 1 previous similar message [ 1933.287664] Lustre: server umount lustre-MDT0000 complete [ 1935.844193] LustreError: 59204:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.862968] LustreError: 59204:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1936.800414] LustreError: 58170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749972 with bad export cookie 3157456951048791205 [ 1936.811380] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1951.199323] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1951.467161] Lustre: server umount lustre-MDT0001 complete [ 1962.619888] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1965.897882] Key type lgssc unregistered [ 1966.163729] LNet: 61908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.177860] LNetError: 61908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.189899] LNet: Removed LNI 192.168.201.126@tcp [ 1967.135653] Key type .llcrypt unregistered [ 1967.142959] Key type ._llcrypt unregistered [ 1989.923369] Key type ._llcrypt registered [ 1989.930636] Key type .llcrypt registered [ 1990.076280] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1991.238964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1991.424135] alg: No test for adler32 (adler32-zlib) [ 1992.539265] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1992.912049] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1994.607213] Key type lgssc registered [ 1995.947096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2004.262672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2004.281742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2005.927153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.691639] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2016.224571] LustreError: 63320:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.244548] LustreError: 63320:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2019.125633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.416388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2023.934677] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2031.753304] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2037.137648] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2043.980297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2044.447707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.788360] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2051.649746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2058.430938] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2076.935781] Lustre: server umount lustre-OST0000 complete [ 2080.736618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2080.755801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2080.774604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.320692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.342830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.441398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.445554] Lustre: Skipped 1 previous similar message [ 2090.626772] Lustre: server umount lustre-MDT0000 complete [ 2094.368826] LustreError: 63299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750130 with bad export cookie 6663414759544478094 [ 2094.383390] LustreError: 63299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2094.383752] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2104.947437] Lustre: server umount lustre-MDT0001 complete [ 2115.888956] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2119.364522] Key type lgssc unregistered [ 2119.674831] LNet: 66887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2119.681254] LNetError: 66887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2119.704735] LNet: Removed LNI 192.168.201.126@tcp [ 2120.847414] Key type .llcrypt unregistered [ 2120.854357] Key type ._llcrypt unregistered [ 2148.147949] Key type ._llcrypt registered [ 2148.149640] Key type .llcrypt registered [ 2148.274974] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2149.308056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2149.320991] alg: No test for adler32 (adler32-zlib) [ 2150.386442] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 2150.624978] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2152.287488] Key type lgssc registered [ 2153.597446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2161.930480] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 23:03:17 (1788750197) [ 2169.725573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2169.782321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.223809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2177.039989] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2181.602869] LustreError: 68590:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2181.657384] LustreError: 68590:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2186.728120] LustreError: 68589:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.682044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.091550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2193.293481] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2201.033752] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2207.414458] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2214.602919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2214.967976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.005906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2221.242778] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2227.657524] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2235.861239] Lustre: Failing over lustre-MDT0000 [ 2236.124423] Lustre: server umount lustre-MDT0000 complete [ 2238.436842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.454652] LustreError: 68590:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.457076] Lustre: Skipped 1 previous similar message [ 2238.512689] LustreError: 68590:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2243.558782] LustreError: 68589:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.577526] LustreError: 68589:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 2247.455938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2247.586877] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2247.716810] Lustre: *** cfs_fail_loc=135, val=0*** [ 2247.839464] LustreError: 71675:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2247.848860] LustreError: 71675:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.126@tcp: cfg command failed: rc = -2 [ 2247.863786] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2247.863786] [ 2247.878214] LustreError: MGC192.168.201.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2247.890051] LustreError: 71663:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2247.895767] LustreError: 71663:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2247.902334] LustreError: 71663:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2247.912719] LustreError: 71663:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2247.934599] Lustre: server umount lustre-MDT0000 complete [ 2247.942137] LustreError: 71663:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2248.677087] LustreError: 68584:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.705936] LustreError: 68584:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 2255.985928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.112906] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.126338] LustreError: 68582:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2256.358434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2256.417449] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2259.878456] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2261.483744] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2261.511372] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2261.561031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2261.953356] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2271.723370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.734046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2271.739312] Lustre: Skipped 2 previous similar messages [ 2271.757842] Lustre: Skipped 1 previous similar message [ 2276.684845] Lustre: server umount lustre-OST0000 complete [ 2281.955813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.956451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2281.980333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2281.991320] Lustre: Skipped 1 previous similar message [ 2286.543437] Lustre: server umount lustre-MDT0000 complete [ 2287.076300] LustreError: 69317:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.086386] LustreError: 69317:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [ 2290.557843] LustreError: 68569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750326 with bad export cookie 12859175477125693843 [ 2290.567936] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2290.573842] LustreError: 68569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2290.818695] Lustre: server umount lustre-MDT0001 complete [ 2301.599735] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2304.835976] Key type lgssc unregistered [ 2305.187198] LNet: 73454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.202781] LNetError: 73454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.222581] LNet: Removed LNI 192.168.201.126@tcp [ 2306.262135] Key type .llcrypt unregistered [ 2306.264248] Key type ._llcrypt unregistered [ 2333.027532] Key type ._llcrypt registered [ 2333.031423] Key type .llcrypt registered [ 2333.150121] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2334.102861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2334.117513] alg: No test for adler32 (adler32-zlib) [ 2335.152251] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 2335.450046] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2337.247440] Key type lgssc registered [ 2338.465736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2346.622192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2346.646141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2348.071826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2352.910109] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2358.243197] LustreError: 74865:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2358.265605] LustreError: 74865:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2362.379667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2362.756046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2367.916752] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2376.578618] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2382.926796] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2390.865593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2391.366236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.394693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2399.270711] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2407.106391] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2414.321321] Lustre: Failing over lustre-MDT0000 [ 2414.594651] Lustre: server umount lustre-MDT0000 complete [ 2415.193117] LustreError: 74861:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.146944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2418.162433] LustreError: 74860:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.171161] Lustre: Skipped 2 previous similar messages [ 2418.218942] LustreError: 74860:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2420.275951] LustreError: 74861:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.915615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.036873] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.200622] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2425.368035] LustreError: 77953:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2425.378616] LustreError: 77953:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.126@tcp: cfg command failed: rc = -9 [ 2425.391295] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2425.391295] [ 2425.396615] LustreError: 77009:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.404793] LustreError: MGC192.168.201.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2425.404811] LustreError: 77942:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2425.404916] LustreError: 77942:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2425.405933] LustreError: 77942:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2425.405952] LustreError: 77942:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2425.415925] LustreError: 75599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750461 with bad export cookie 5888465784523816244 [ 2425.431882] LustreError: 77009:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 2425.448022] Lustre: server umount lustre-MDT0000 complete [ 2425.454332] LustreError: 75599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2425.525637] LustreError: 77942:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2432.166109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.566343] LustreError: 74865:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.594148] LustreError: 74865:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 9 previous similar messages [ 2435.922276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2435.994722] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2440.748549] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2441.220200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2441.241876] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2441.296252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2441.788691] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2451.425513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.430315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2451.441896] Lustre: Skipped 1 previous similar message [ 2451.465682] Lustre: Skipped 1 previous similar message [ 2456.549716] LustreError: 76688:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.562583] LustreError: 76688:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2456.612943] Lustre: server umount lustre-OST0000 complete [ 2461.665039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2461.666081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.676471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2461.682375] Lustre: Skipped 1 previous similar message [ 2461.698830] Lustre: Skipped 1 previous similar message [ 2466.966967] Lustre: server umount lustre-MDT0000 complete [ 2471.046969] LustreError: 75599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750507 with bad export cookie 5888465784523816405 [ 2471.052249] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2471.054395] LustreError: 75599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2471.065132] LustreError: Skipped 1 previous similar message [ 2471.326948] Lustre: server umount lustre-MDT0001 complete [ 2481.864873] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2484.583895] Key type lgssc unregistered [ 2484.959363] LNet: 79735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.986992] LNetError: 79735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2485.016557] LNet: Removed LNI 192.168.201.126@tcp [ 2486.049724] Key type .llcrypt unregistered [ 2486.051897] Key type ._llcrypt unregistered [ 2501.679405] Key type ._llcrypt registered [ 2501.681875] Key type .llcrypt registered [ 2501.817955] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2502.154315] Key type .llcrypt unregistered [ 2502.159950] Key type ._llcrypt unregistered [ 2526.835341] Key type ._llcrypt registered [ 2526.837669] Key type .llcrypt registered [ 2526.985446] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2528.859789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2528.897981] alg: No test for adler32 (adler32-zlib) [ 2530.120899] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 2530.502448] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2532.311228] Key type lgssc registered [ 2533.932620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2543.433590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2543.453041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.895941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.926311] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2555.371588] LustreError: 81635:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.409428] LustreError: 81635:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2559.956933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2560.383457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2565.893954] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2574.311414] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2580.403680] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2588.423092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2589.036899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.088548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2597.305571] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2604.898426] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2615.780489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.806516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2615.810269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2617.379977] Lustre: server umount lustre-OST0000 complete [ 2621.920613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2621.935205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.945804] Lustre: Skipped 1 previous similar message [ 2621.950726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2621.956052] Lustre: Skipped 1 previous similar message [ 2626.018847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2626.021732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2627.576532] Lustre: server umount lustre-MDT0000 complete [ 2631.146741] LustreError: 81635:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.158566] LustreError: 81635:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2631.175576] LustreError: 81617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750667 with bad export cookie 3675236139581052669 [ 2631.184285] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2631.610643] Lustre: server umount lustre-MDT0001 complete [ 2643.685620] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2646.857481] Key type lgssc unregistered [ 2647.234758] LNet: 85167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2647.242497] LNetError: 85167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2647.292644] LNet: Removed LNI 192.168.201.126@tcp [ 2648.381737] Key type .llcrypt unregistered [ 2648.383241] Key type ._llcrypt unregistered [ 2673.507706] Key type ._llcrypt registered [ 2673.509288] Key type .llcrypt registered [ 2673.597962] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2674.724027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2674.823273] alg: No test for adler32 (adler32-zlib) [ 2676.032535] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 2676.342288] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2678.080818] Key type lgssc registered [ 2679.548723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2687.695757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2687.728389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.185554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.556867] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2699.754654] LustreError: 86577:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.768303] LustreError: 86577:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2702.014914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.425808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2707.239399] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2714.983793] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2720.795601] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2727.318409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2727.671885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.840739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2735.253333] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2744.038625] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2755.552501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2755.560544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.581482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2757.603258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2757.610708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.611271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2760.750353] Lustre: server umount lustre-OST0000 complete [ 2767.842496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.853408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2767.861418] Lustre: Skipped 1 previous similar message [ 2767.872550] Lustre: Skipped 2 previous similar messages [ 2771.466603] Lustre: server umount lustre-MDT0000 complete [ 2772.959852] LustreError: 87831:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2772.981341] LustreError: 87831:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2775.400031] LustreError: 86556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750811 with bad export cookie 7503800944114166800 [ 2775.410628] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2775.781475] Lustre: server umount lustre-MDT0001 complete [ 2787.581540] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2790.054280] Key type lgssc unregistered [ 2790.426782] LNet: 90111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2790.434674] LNetError: 90111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2790.444480] LNet: Removed LNI 192.168.201.126@tcp [ 2791.635744] Key type .llcrypt unregistered [ 2791.638555] Key type ._llcrypt unregistered [ 2817.473295] Key type ._llcrypt registered [ 2817.476953] Key type .llcrypt registered [ 2817.615287] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2818.868033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2819.051600] alg: No test for adler32 (adler32-zlib) [ 2820.262335] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 2820.594992] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2822.327287] Key type lgssc registered [ 2823.691254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2832.354405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2832.368978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2833.887566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2839.079772] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2843.109439] LustreError: 91522:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2843.128656] LustreError: 91522:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2848.227053] LustreError: 91523:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2848.902374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2849.235811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2854.740673] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2862.839649] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2869.261482] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2876.941544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2877.551684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.580799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2884.811985] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2892.393217] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2904.544882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2904.566355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.571804] LustreError: Skipped 1 previous similar message [ 2904.572461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2904.589789] Lustre: Skipped 1 previous similar message [ 2904.620222] Lustre: Skipped 1 previous similar message [ 2909.684634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2909.698106] Lustre: Skipped 1 previous similar message [ 2914.788240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2918.367146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2918.623101] Lustre: server umount lustre-OST0000 complete [ 2925.027033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2925.040348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2925.046408] Lustre: Skipped 1 previous similar message [ 2925.059449] Lustre: Skipped 2 previous similar messages [ 2930.148056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2937.311926] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2937.591511] Lustre: server umount lustre-MDT0000 complete [ 2940.391217] LustreError: 91522:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.428637] LustreError: 91522:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2941.365732] LustreError: 91502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750977 with bad export cookie 899037613314036812 [ 2941.367641] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2941.383328] LustreError: 91502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2941.832706] Lustre: server umount lustre-MDT0001 complete [ 2953.502112] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2956.917924] Key type lgssc unregistered [ 2957.280376] LNet: 95059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2957.293170] LNetError: 95059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2957.315958] LNet: Removed LNI 192.168.201.126@tcp [ 2958.502971] Key type .llcrypt unregistered [ 2958.511380] Key type ._llcrypt unregistered [ 2982.957462] Key type ._llcrypt registered [ 2982.961701] Key type .llcrypt registered [ 2983.140328] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2984.592484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2984.703536] alg: No test for adler32 (adler32-zlib) [ 2985.908098] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 2986.244799] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2987.927279] Key type lgssc registered [ 2989.464210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2998.661335] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2998.696505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3014.303514] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3019.423358] LustreError: 96443:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 3046.175499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3052.957984] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3061.660095] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3067.442125] Lustre: server umount lustre-OST0000 complete [ 3077.281826] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 23:18:32 (1788751112) [ 3085.227582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.742583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3090.915394] LustreError: 98081:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.929266] LustreError: 98081:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3091.213866] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3095.009759] LustreError: 98082:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.131121] LustreError: 98081:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.611711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.082096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3105.762450] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3113.067831] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3119.392716] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3126.208048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3126.540286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3130.669148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3132.810718] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3140.522807] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3153.376779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3153.389754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.418749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3156.449296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3156.460365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3156.462724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.482211] Lustre: Skipped 1 previous similar message [ 3157.650534] Lustre: server umount lustre-OST0000 complete [ 3161.570579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.587046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3166.692813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3166.701027] Lustre: Skipped 2 previous similar messages [ 3167.702735] Lustre: server umount lustre-MDT0000 complete [ 3171.304230] LustreError: 98816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751207 with bad export cookie 14823564292589900043 [ 3171.311345] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3171.313756] LustreError: 98816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3171.606802] Lustre: server umount lustre-MDT0001 complete [ 3183.597879] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3186.425627] Key type lgssc unregistered [ 3186.680273] LNet: 101772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3186.684720] LNetError: 101772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3186.696580] LNet: Removed LNI 192.168.201.126@tcp [ 3187.487528] Key type .llcrypt unregistered [ 3187.488750] Key type ._llcrypt unregistered [ 3230.447164] Key type ._llcrypt registered [ 3230.449532] Key type .llcrypt registered [ 3230.610289] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3231.989168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3232.302877] alg: No test for adler32 (adler32-zlib) [ 3233.642881] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 3233.872367] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3235.583263] Key type lgssc registered [ 3236.933653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3245.271703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3245.310518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.896438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.326455] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3256.305987] LustreError: 103959:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3256.332597] LustreError: 103959:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3259.413793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.863579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3264.758600] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3272.871731] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3279.233418] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3286.002530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3286.407478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3291.585277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3292.881771] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3300.780487] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3314.150337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.164875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3314.165372] Lustre: Skipped 1 previous similar message [ 3314.165889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3314.193284] Lustre: Skipped 1 previous similar message [ 3319.206658] Lustre: server umount lustre-OST0000 complete [ 3324.391357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.411732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3329.506383] LustreError: 105378:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3329.517767] LustreError: 105378:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3329.535272] Lustre: server umount lustre-MDT0000 complete [ 3333.031150] LustreError: 103983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751369 with bad export cookie 9076066589960501758 [ 3333.034706] LustreError: 103983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3333.035465] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3333.294094] Lustre: server umount lustre-MDT0001 complete [ 3344.194469] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3347.123365] Key type lgssc unregistered [ 3347.389745] LNet: 107505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3347.403500] LNetError: 107505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3347.418300] LNet: Removed LNI 192.168.201.126@tcp [ 3348.326377] Key type .llcrypt unregistered [ 3348.328268] Key type ._llcrypt unregistered [ 3366.655909] Key type ._llcrypt registered [ 3366.657377] Key type .llcrypt registered [ 3366.718776] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3367.620039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3367.685491] alg: No test for adler32 (adler32-zlib) [ 3368.649395] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 3368.920306] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3370.623226] Key type lgssc registered [ 3371.518158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3377.433518] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3377.453246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3392.993173] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3398.111321] LustreError: 108743:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 3424.801405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.337251] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3436.477835] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3443.039069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3443.306706] LustreError: MGC192.168.201.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3443.314755] LustreError: 109766:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3443.325822] LustreError: 109766:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3443.339779] LustreError: 109766:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3443.420386] Lustre: server umount lustre-MDT0000 complete [ 3443.422517] LustreError: 109766:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3464.159167] Lustre: 108755:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788751484/real 1788751484] req@00000000aefe5683 x1875641793252736/t0(0) o101->MGC192.168.201.126@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1788751500 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3464.188933] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3464.197828] LustreError: 108755:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3484.751448] Lustre: server umount lustre-OST0000 complete [ 3494.108215] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 3503.718898] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3535.690096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3535.886067] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3535.904163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3535.970681] Lustre: lustre-MDT0000: new disk, initializing [ 3536.032711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.049232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3540.158481] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3549.038978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3549.233725] Lustre: 112559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3549.304783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3549.310656] Lustre: Skipped 1 previous similar message [ 3549.426284] Lustre: lustre-MDT0001: new disk, initializing [ 3549.547511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3549.588576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3549.613476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3553.660444] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3562.080497] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3567.378706] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3573.334433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3573.547447] Lustre: lustre-OST0000: new disk, initializing [ 3573.550851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3573.555505] Lustre: 114527:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3573.619421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3579.012740] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3581.056531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3581.076543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3581.255621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3587.676634] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3591.729838] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3591.967634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3596.365834] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3596.565904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3601.894407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3601.898348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3601.901535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3601.902045] LustreError: Skipped 1 previous similar message [ 3601.943025] Lustre: Skipped 1 previous similar message [ 3604.797535] Lustre: server umount lustre-OST0000 complete [ 3611.104328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3611.111458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3611.121666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3611.124623] Lustre: Skipped 1 previous similar message [ 3612.131192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3612.132932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3612.159882] Lustre: Skipped 1 previous similar message [ 3617.250860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3617.253345] Lustre: Skipped 1 previous similar message [ 3622.368310] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3622.373183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3622.383515] Lustre: Skipped 1 previous similar message [ 3622.550771] Lustre: server umount lustre-MDT0000 complete [ 3626.201538] LustreError: 112553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751662 with bad export cookie 8141376550609029571 [ 3626.218137] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3626.548585] Lustre: server umount lustre-MDT0001 complete [ 3634.744127] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 23:27:50 (1788751670) [ 3638.012802] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3652.741481] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 3663.085924] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3694.500315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3694.771360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3694.794719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3694.861475] Lustre: lustre-MDT0000: new disk, initializing [ 3694.908221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3694.923378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3698.995408] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3708.255155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.318358] Lustre: 119425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3708.386600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3708.389806] Lustre: Skipped 1 previous similar message [ 3708.453528] Lustre: lustre-MDT0001: new disk, initializing [ 3708.529752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3708.576697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3708.593634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3712.396367] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3720.896733] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3726.475757] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3732.972500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3733.204675] Lustre: lustre-OST0000: new disk, initializing [ 3733.208247] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3733.216440] Lustre: 121392:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3733.341187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3739.103296] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3742.340430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3742.354249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3742.488250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3748.361450] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3752.462384] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3752.693412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3757.227402] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3757.525439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3763.177032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.183663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3763.191608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3763.194716] LustreError: Skipped 1 previous similar message [ 3763.215395] Lustre: Skipped 1 previous similar message [ 3765.838514] Lustre: server umount lustre-OST0000 complete [ 3769.827187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3769.834180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3769.859623] Lustre: Skipped 1 previous similar message [ 3774.975293] Lustre: server umount lustre-MDT0000 complete [ 3778.353646] LustreError: 121391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751814 with bad export cookie 8141376550609030453 [ 3778.356041] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3778.360053] LustreError: 121391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3778.598987] Lustre: server umount lustre-MDT0001 complete [ 3785.951965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3786.378788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3790.540573] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3791.841568] LustreError: 123615:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.852360] LustreError: 123615:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3795.943745] LustreError: 123616:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3798.766584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3803.290511] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3809.894736] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3815.138956] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3821.287035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3828.061866] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3835.665359] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3841.766980] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3845.601485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3845.604679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3845.627714] Lustre: Skipped 1 previous similar message [ 3845.635504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3845.645024] Lustre: Skipped 3 previous similar messages [ 3850.912594] Lustre: server umount lustre-OST0000 complete [ 3855.339352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3855.346131] LustreError: Skipped 1 previous similar message [ 3860.448842] Lustre: server umount lustre-MDT0000 complete [ 3861.995157] LustreError: 124345:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3862.021218] LustreError: 124345:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3863.955924] LustreError: 123596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751900 with bad export cookie 8141376550609031328 [ 3863.965076] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3863.965462] LustreError: 123596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3864.346363] Lustre: server umount lustre-MDT0001 complete [ 3874.799311] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3877.234586] Key type lgssc unregistered [ 3877.451500] LNet: 127242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3877.454206] LNetError: 127242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3877.481874] LNet: Removed LNI 192.168.201.126@tcp [ 3878.439529] Key type .llcrypt unregistered [ 3878.442720] Key type ._llcrypt unregistered [ 3894.652643] Key type ._llcrypt registered [ 3894.654662] Key type .llcrypt registered [ 3894.777053] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 3909.215940] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3910.539357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3910.568471] alg: No test for adler32 (adler32-zlib) [ 3911.612393] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 3911.838710] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3913.527204] Key type lgssc registered [ 3914.609809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3948.789546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3948.830456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3950.020313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3950.038616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3950.117772] Lustre: lustre-MDT0000: new disk, initializing [ 3950.183765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3950.204777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3954.222509] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3963.801868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.923069] Lustre: 130390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3963.969663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3963.972687] Lustre: Skipped 1 previous similar message [ 3964.023550] Lustre: lustre-MDT0001: new disk, initializing [ 3964.097410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3964.150394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3964.158288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3968.158479] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3976.581655] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3981.296449] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3987.202053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3987.413743] Lustre: lustre-OST0000: new disk, initializing [ 3987.417314] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3987.424837] Lustre: 132359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3987.508811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3993.064625] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3993.677761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3993.690784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3993.772209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4000.761767] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4004.943955] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4005.113207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4008.912689] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4009.096441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4014.560356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4014.568323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4014.571040] LustreError: Skipped 1 previous similar message [ 4014.573808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4014.579874] Lustre: Skipped 1 previous similar message [ 4014.600165] Lustre: Skipped 1 previous similar message [ 4016.912638] Lustre: server umount lustre-OST0000 complete [ 4020.705650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4020.714564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4020.735242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4024.802184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4024.806868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4028.528350] Lustre: server umount lustre-MDT0000 complete [ 4031.887327] LustreError: 130385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752068 with bad export cookie 11022603715348529948 [ 4031.895512] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4031.897207] LustreError: 130385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4032.193112] Lustre: server umount lustre-MDT0001 complete [ 4039.176421] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 23:34:34 (1788752074) [ 4044.300262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4044.573694] LustreError: 134771:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4044.588606] LustreError: 134771:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 4044.633761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4048.172882] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4049.890903] LustreError: 134772:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4053.987644] LustreError: 134771:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4056.469936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4056.779374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4061.162256] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4068.140385] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4073.946375] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4077.540074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4077.556736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4077.587772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4077.595713] Lustre: Skipped 1 previous similar message [ 4081.636422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4081.650484] Lustre: Skipped 3 previous similar messages [ 4081.811396] Lustre: server umount lustre-MDT0000 complete [ 4085.203479] LustreError: 134753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752121 with bad export cookie 11022603715348530830 [ 4085.207821] LustreError: 134753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4085.213678] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4085.476804] Lustre: server umount lustre-MDT0001 complete [ 4092.595183] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 23:35:28 (1788752128) [ 4098.209837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4113.761414] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4118.879498] LustreError: 137188:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 4144.610952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4150.433028] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4157.063614] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4158.829137] Lustre: server umount lustre-OST0000 complete [ 4166.945710] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 23:36:42 (1788752202) [ 4172.795790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4173.189029] LustreError: 138632:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4173.265110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4177.591852] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4178.405485] LustreError: 138633:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4185.758607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4190.137480] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4196.493546] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4200.828483] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4205.374153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4205.680846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4205.687910] Lustre: Skipped 1 previous similar message [ 4210.428989] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4217.193841] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4227.044963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4227.051774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4227.060914] Lustre: Skipped 1 previous similar message [ 4227.065225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4232.767965] Lustre: server umount lustre-MDT0000 complete [ 4233.696508] LustreError: 138651:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.708940] LustreError: 138651:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 4235.556128] LustreError: 139367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752271 with bad export cookie 11022603715348531495 [ 4235.558421] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4235.564412] LustreError: 139367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4235.892460] Lustre: server umount lustre-MDT0001 complete [ 4248.008536] Lustre: server umount lustre-OST0000 complete [ 4253.434725] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 23:38:09 (1788752289) [ 4257.674983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4258.040936] LustreError: 142284:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4261.477923] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4267.366761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4271.071726] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4276.834446] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4280.781555] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4285.877917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4286.112733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4286.116975] Lustre: Skipped 2 previous similar messages [ 4291.152655] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4296.350431] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4299.362635] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4303.531209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4303.716830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4306.001924] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4306.142188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4308.963372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4308.964026] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4308.965563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4308.965569] Lustre: Skipped 3 previous similar messages [ 4308.971265] LustreError: Skipped 1 previous similar message [ 4308.988497] Lustre: Skipped 3 previous similar messages [ 4309.380433] Lustre: server umount lustre-OST0000 complete [ 4317.900919] Lustre: server umount lustre-MDT0000 complete [ 4319.203438] LustreError: 142280:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4319.212764] LustreError: 142280:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 4320.481630] LustreError: 142266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752356 with bad export cookie 11022603715348532566 [ 4320.489179] LustreError: 142266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4320.496044] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4320.703180] Lustre: server umount lustre-MDT0001 complete [ 4325.971280] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 23:39:21 (1788752361) [ 4330.646741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4346.143545] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4351.263411] LustreError: 146466:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 4382.541644] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4389.235158] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4394.933280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4395.270831] LustreError: 146491:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4395.426590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4399.370349] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4406.352331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4410.431868] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4416.627918] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4421.472712] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4425.671078] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4425.886395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4429.535953] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4429.715636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4432.358365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.363032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4432.375212] Lustre: Skipped 2 previous similar messages [ 4432.376042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4432.383316] LustreError: Skipped 1 previous similar message [ 4432.392461] Lustre: Skipped 5 previous similar messages [ 4437.438903] Lustre: server umount lustre-MDT0000 complete [ 4440.230636] LustreError: 146472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752476 with bad export cookie 11022603715348533588 [ 4440.234717] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4440.240831] LustreError: 146472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4440.531485] Lustre: server umount lustre-MDT0001 complete [ 4453.157451] Lustre: server umount lustre-OST0000 complete [ 4460.038673] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 23:41:35 (1788752495) [ 4464.976190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4480.480427] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4485.599806] LustreError: 150658:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 4511.327406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4511.330027] Lustre: Skipped 3 previous similar messages [ 4516.879309] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4523.066367] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4528.651598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4528.953128] LustreError: 150683:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.962100] LustreError: 150683:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 4529.026129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4531.979779] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4543.346455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4551.269454] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4561.685920] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4569.772443] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4579.452960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4579.711170] Lustre: lustre-OST0001: new disk, initializing [ 4579.719181] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4579.727178] Lustre: 153533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4589.083766] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4589.692074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4589.709955] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4589.842189] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4600.120599] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4606.016206] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4606.172820] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4611.554192] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4611.713819] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4615.664848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4615.671947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4615.686074] Lustre: Skipped 3 previous similar messages [ 4615.696391] Lustre: Skipped 1 previous similar message [ 4619.891618] Lustre: server umount lustre-OST0000 complete [ 4625.895518] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4625.903362] LustreError: Skipped 1 previous similar message [ 4638.175708] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4638.502650] Lustre: server umount lustre-OST0001 complete [ 4649.547072] Lustre: server umount lustre-MDT0000 complete [ 4653.700028] LustreError: 150665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752689 with bad export cookie 11022603715348534540 [ 4653.707251] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4653.724716] LustreError: 150665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4654.118879] Lustre: server umount lustre-MDT0001 complete [ 4677.972902] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 23:45:13 (1788752713) [ 4679.836227] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4681.816042] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4683.665994] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 23:45:18 (1788752718) [ 4691.284529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4691.527821] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4691.571991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4697.102242] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4706.247411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4706.395745] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4706.449274] Lustre: 157093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4712.795850] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4721.173594] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4727.627610] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4735.037542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4735.213886] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4742.211130] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4745.396027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4750.676800] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4756.237807] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4756.483789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4761.408965] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4761.708131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4766.181874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4766.189766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4766.203087] LustreError: Skipped 1 previous similar message [ 4766.215201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4766.234935] Lustre: Skipped 5 previous similar messages [ 4766.269193] Lustre: Skipped 9 previous similar messages [ 4769.665694] Lustre: server umount lustre-OST0000 complete [ 4791.997463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4792.417387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4792.420294] Lustre: Skipped 6 previous similar messages [ 4798.852310] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4806.309531] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4806.788700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4806.809918] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4806.819815] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4806.829214] Lustre: Skipped 1 previous similar message [ 4811.826708] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4812.115891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4816.835914] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4817.087986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4823.115733] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff937e89ea0000.ost_server_uuid 50 [ 4824.539661] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff937e89ea0000.ost_server_uuid in FULL state after 0 sec [ 4824.740165] LustreError: 157100:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.26@tcp) returned error from blocking AST (req@00000000d12c26c6 x1875642362881152 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000005f28719c/0x98f827adc9401e66 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.26@tcp remote: 0x53333d19ae820352 expref: 5 pid: 157100 timeout: 4924 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4824.779687] LustreError: lustre-MDT0000: A client on nid 192.168.201.26@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4824.791772] LustreError: 157090:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.26@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000005f28719c/0x98f827adc9401e66 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.26@tcp remote: 0x53333d19ae820352 expref: 6 pid: 157100 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4831.204231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4834.743859] Lustre: server umount lustre-OST0000 complete [ 4844.528182] Lustre: server umount lustre-MDT0000 complete [ 4847.883057] LustreError: 157087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752884 with bad export cookie 11022603715348535723 [ 4847.894331] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4847.900301] LustreError: 157087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4854.491279] Lustre: server umount lustre-MDT0001 complete [ 4866.835757] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 4869.610847] Key type lgssc unregistered [ 4869.901754] LNet: 163205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4869.909776] LNetError: 163205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4869.957320] LNet: Removed LNI 192.168.201.126@tcp [ 4871.117246] Key type .llcrypt unregistered [ 4871.118883] Key type ._llcrypt unregistered [ 4895.665697] Key type ._llcrypt registered [ 4895.668546] Key type .llcrypt registered [ 4895.769804] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 4897.114795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4897.142798] alg: No test for adler32 (adler32-zlib) [ 4898.204494] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 4898.438463] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 4900.095215] Key type lgssc registered [ 4901.121545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4908.330726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4908.365047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4909.862564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4914.103127] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4920.293728] LustreError: 164615:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4920.311083] LustreError: 164615:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 4922.210811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4922.557349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4926.674328] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4933.662066] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4939.245536] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4945.794361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4946.038246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4951.634235] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4955.274986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4958.360792] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4964.619194] Lustre: Failing over lustre-MDT0000 [ 4964.888601] Lustre: server umount lustre-MDT0000 complete [ 4965.860712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4965.866929] LustreError: 167107:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4965.872459] Lustre: Skipped 2 previous similar messages [ 4965.908454] LustreError: 167107:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 4970.977183] LustreError: 164616:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4970.991581] LustreError: 164616:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 4976.099510] LustreError: 165346:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.140331] LustreError: 165346:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 4977.085971] LustreError: 165353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.26@tcp arrived at 1788753013 with bad export cookie 17020068004936765061 [ 4977.100145] LustreError: 165353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4981.218455] LustreError: 164615:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4981.236149] LustreError: 164615:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 4982.240133] Lustre: 163926:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788753002/real 1788753002] req@00000000cc809a05 x1875643396547840/t0(0) o400->MGC192.168.201.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788753018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4982.271921] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4984.495362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4992.479857] LustreError: 163924:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009e8b5ca3 x1875643396553344/t0(0) o250->MGC192.168.201.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4992.506509] LustreError: 164616:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4992.528138] LustreError: 164616:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 8 previous similar messages [ 4992.767405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4992.877204] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4997.816872] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4998.123955] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4998.138642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5000.174793] Lustre: lustre-MDT0000: Denying connection for new client 4e18696b-6944-4ccd-bb36-9c84a7796b7d (at 192.168.201.26@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5008.354829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.378051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5008.386288] Lustre: Skipped 1 previous similar message [ 5013.479871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5014.678335] Lustre: server umount lustre-OST0000 complete [ 5018.545639] LustreError: 168347:0:(ldlm_lib.c:3042:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5018.560468] Lustre: 167695:0:(ldlm_lib.c:2442:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5018.564229] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5018.579339] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5018.589780] LustreError: 167695:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a5a08bfc x1875643396567808/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5018.603421] LustreError: 167695:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5018.606110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5018.616904] LustreError: 167695:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5018.622296] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5018.626238] Lustre: Skipped 1 previous similar message [ 5018.628932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5018.628940] Lustre: Skipped 1 previous similar message [ 5018.987024] Lustre: server umount lustre-MDT0000 complete [ 5022.497822] LustreError: 164597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753058 with bad export cookie 17020068004936765159 [ 5022.512315] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5022.516304] LustreError: 164597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5022.517551] LustreError: 165346:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5024.921446] Lustre: server umount lustre-MDT0001 complete [ 5034.650283] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5037.278329] Key type lgssc unregistered [ 5037.595333] LNet: 169098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5037.606150] LNetError: 169098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5037.629649] LNet: Removed LNI 192.168.201.126@tcp [ 5038.508184] Key type .llcrypt unregistered [ 5038.510479] Key type ._llcrypt unregistered [ 5059.220453] Key type ._llcrypt registered [ 5059.224521] Key type .llcrypt registered [ 5059.316604] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5060.698653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5060.730738] alg: No test for adler32 (adler32-zlib) [ 5061.773798] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 5062.012168] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 5063.695763] Key type lgssc registered [ 5064.652897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5070.985393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5071.003481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5072.432026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5076.319513] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5082.594395] LustreError: 170485:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5082.615944] LustreError: 170485:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5083.594699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5083.890277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5087.757200] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5093.572563] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5098.328951] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5104.768379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5105.130492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5111.456872] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5117.291809] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5117.443398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5124.578355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5124.591540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5124.600467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5127.652300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5127.655077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5127.676396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5129.582967] Lustre: server umount lustre-OST0000 complete [ 5132.769072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5132.776144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5132.779343] Lustre: Skipped 1 previous similar message [ 5132.790963] Lustre: Skipped 1 previous similar message [ 5137.889557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5137.894525] Lustre: Skipped 2 previous similar messages [ 5138.954200] Lustre: server umount lustre-MDT0000 complete [ 5142.423821] LustreError: 170465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753178 with bad export cookie 16128282556093230485 [ 5142.432459] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5142.437418] LustreError: 170465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5142.708613] Lustre: server umount lustre-MDT0001 complete [ 5152.796160] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5157.400493] Key type lgssc unregistered [ 5158.714093] LNet: 174015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5158.738901] LNetError: 174015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5158.801361] LNet: Removed LNI 192.168.201.126@tcp [ 5160.836255] Key type .llcrypt unregistered [ 5160.837910] Key type ._llcrypt unregistered [ 5202.649966] Key type ._llcrypt registered [ 5202.658109] Key type .llcrypt registered [ 5202.855825] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5203.983907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5203.995490] alg: No test for adler32 (adler32-zlib) [ 5205.099133] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 5205.312896] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 5206.991207] Key type lgssc registered [ 5207.888532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5215.474374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5215.501783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.960812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5221.511802] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5227.502079] LustreError: 175718:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5227.523156] LustreError: 175718:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5230.550824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5230.716983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5236.695623] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5246.333217] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5252.421575] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5259.312543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5259.767631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5262.855148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5266.904824] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5274.295909] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5283.298713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5283.312245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.337803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5285.348074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5285.353304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5285.356056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.954608] Lustre: server umount lustre-OST0000 complete [ 5295.587988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5295.592177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5295.608362] Lustre: Skipped 1 previous similar message [ 5295.624375] Lustre: Skipped 2 previous similar messages [ 5298.795825] Lustre: server umount lustre-MDT0000 complete [ 5300.706157] LustreError: 175714:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5300.728467] LustreError: 175714:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5302.579315] LustreError: 175696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753338 with bad export cookie 12696272242124453608 [ 5302.583445] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5302.607917] LustreError: 175696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5303.240252] Lustre: server umount lustre-MDT0001 complete [ 5314.775558] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5317.781945] Key type lgssc unregistered [ 5318.044821] LNet: 179250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5318.049982] LNetError: 179250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5318.073387] LNet: Removed LNI 192.168.201.126@tcp [ 5319.043523] Key type .llcrypt unregistered [ 5319.045420] Key type ._llcrypt unregistered [ 5341.331144] Key type ._llcrypt registered [ 5341.332399] Key type .llcrypt registered [ 5341.409391] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5342.624305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5342.678375] alg: No test for adler32 (adler32-zlib) [ 5343.798768] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 5344.135818] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 5345.807272] Key type lgssc registered [ 5347.447554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5359.222770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5359.239739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5360.576589] Lustre: *** cfs_fail_loc=135, val=0*** [ 5360.649978] LustreError: 180769:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5360.662420] LustreError: 180769:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.126@tcp: cfg command failed: rc = -2 [ 5360.667969] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5360.667969] [ 5360.677786] LustreError: MGC192.168.201.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5360.689924] LustreError: 180731:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5360.696908] LustreError: 180731:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5360.706417] LustreError: 180731:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5360.714620] LustreError: 180731:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5360.826915] Lustre: server umount lustre-MDT0000 complete [ 5360.831219] LustreError: 180731:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5366.540223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5366.984259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5371.647750] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5377.508481] LustreError: 181141:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5377.524603] LustreError: 181141:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5378.531737] LustreError: 181140:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5381.557039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5382.046314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5388.178833] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5396.415601] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5403.023258] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5407.138063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5407.144904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5407.149726] Lustre: Skipped 1 previous similar message [ 5407.159610] Lustre: Skipped 1 previous similar message [ 5411.424375] Lustre: server umount lustre-MDT0000 complete [ 5412.322478] LustreError: 181141:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5412.359711] LustreError: 181141:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5414.936766] LustreError: 181120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753451 with bad export cookie 5058615550873727516 [ 5414.943594] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5414.944922] LustreError: 181120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5415.366725] Lustre: server umount lustre-MDT0001 complete [ 5424.476873] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5426.587927] Key type lgssc unregistered [ 5426.807591] LNet: 183460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5426.812412] LNetError: 183460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5427.879468] LNet: Removed LNI 192.168.201.126@tcp [ 5428.706668] Key type .llcrypt unregistered [ 5428.708716] Key type ._llcrypt unregistered [ 5452.868906] Key type ._llcrypt registered [ 5452.871326] Key type .llcrypt registered [ 5453.003139] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5453.295401] Key type .llcrypt unregistered [ 5453.297289] Key type ._llcrypt unregistered [ 5474.042242] Key type ._llcrypt registered [ 5474.044498] Key type .llcrypt registered [ 5474.204955] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5475.280306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5475.348938] alg: No test for adler32 (adler32-zlib) [ 5476.623239] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 5477.044050] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 5478.767347] Key type lgssc registered [ 5480.133778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5489.011813] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5489.046852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5504.543449] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5509.672481] LustreError: 185588:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 5536.351237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5542.956895] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5550.716570] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5557.537523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5558.056211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5558.142457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5562.724273] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5563.367872] LustreError: 185614:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5563.374126] LustreError: 185614:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 5568.487567] LustreError: 185613:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5571.430973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5571.564441] LustreError: 186625:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5571.593639] LustreError: 186625:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5571.868205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5576.722502] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5584.871850] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5591.120421] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5593.962164] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5607.905069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5607.913049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5607.916017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5607.921265] Lustre: Skipped 1 previous similar message [ 5607.932705] Lustre: Skipped 1 previous similar message [ 5612.230444] Lustre: server umount lustre-OST0000 complete [ 5618.145507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5618.149084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5618.171902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5618.188808] Lustre: Skipped 1 previous similar message [ 5621.749492] Lustre: server umount lustre-MDT0000 complete [ 5623.271742] LustreError: 187987:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5623.291933] LustreError: 187987:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5626.248884] LustreError: 185594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753662 with bad export cookie 2934414542105975645 [ 5626.261404] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5626.271560] LustreError: 185594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5627.019306] Lustre: server umount lustre-MDT0001 complete [ 5638.816292] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5641.488439] Key type lgssc unregistered [ 5641.769805] LNet: 189614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5641.780497] LNetError: 189614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5641.791968] LNet: Removed LNI 192.168.201.126@tcp [ 5642.612735] Key type .llcrypt unregistered [ 5642.614720] Key type ._llcrypt unregistered [ 5666.389336] Key type ._llcrypt registered [ 5666.390838] Key type .llcrypt registered [ 5666.523215] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5667.975158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5668.223104] alg: No test for adler32 (adler32-zlib) [ 5669.343310] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 5669.553233] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 5671.239630] Key type lgssc registered [ 5672.382304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5680.500480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5680.513856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5681.958973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5685.667267] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5692.386621] LustreError: 191024:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5692.417767] LustreError: 191024:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5694.622286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5694.862701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5699.925451] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5708.035175] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5713.798873] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5720.932313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5721.388071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5727.567364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5728.029720] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5735.759226] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5740.021532] Lustre: Failing over lustre-MDT0000 [ 5740.305303] Lustre: server umount lustre-MDT0000 complete [ 5741.024064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5741.037705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5741.074120] LustreError: 191751:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.676210] LustreError: 191020:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.133973] LustreError: 191025:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.144948] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5750.243048] LustreError: 191021:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5750.258444] LustreError: 191021:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 5759.088767] LustreError: 193767:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.119424] LustreError: 193767:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 5759.388545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5759.543924] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5759.824699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5759.888980] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5763.119254] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5763.646781] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5765.111624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5765.156066] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5765.232523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5767.728140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5771.289843] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5784.056593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5784.076073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5784.090628] Lustre: Skipped 1 previous similar message [ 5784.106506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5784.761980] Lustre: server umount lustre-OST0000 complete [ 5790.688295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5790.698943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5790.701050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5790.722470] Lustre: Skipped 1 previous similar message [ 5790.730733] Lustre: Skipped 1 previous similar message [ 5794.347241] Lustre: server umount lustre-MDT0000 complete [ 5795.813165] LustreError: 191025:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5795.830223] LustreError: 191025:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5798.087145] LustreError: 191006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753834 with bad export cookie 11455997408585160443 [ 5798.099590] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5808.755562] Lustre: server umount lustre-MDT0001 complete [ 5819.254349] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5821.989145] Key type lgssc unregistered [ 5822.265907] LNet: 195809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5822.275943] LNetError: 195809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5823.336637] LNet: Removed LNI 192.168.201.126@tcp [ 5824.140532] Key type .llcrypt unregistered [ 5824.143616] Key type ._llcrypt unregistered [ 5846.770053] Key type ._llcrypt registered [ 5846.770906] Key type .llcrypt registered [ 5846.852626] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5848.266477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5848.286351] alg: No test for adler32 (adler32-zlib) [ 5849.399423] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 5849.634141] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 5851.279168] Key type lgssc registered [ 5853.156274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5862.878619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5862.891971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5864.402694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5868.992350] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5874.660596] LustreError: 197218:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5874.686579] LustreError: 197218:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5877.495150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.915413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5883.419372] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5891.812584] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5898.305569] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5905.400277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5905.644546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5909.746510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5910.890636] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5920.461651] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5923.940055] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5928.810428] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5945.401107] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5952.993646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5953.000690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5953.005109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5953.024742] Lustre: Skipped 1 previous similar message [ 5957.907421] Lustre: server umount lustre-OST0000 complete [ 5963.233598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5963.235318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5963.263778] Lustre: Skipped 1 previous similar message [ 5963.280508] Lustre: Skipped 2 previous similar messages [ 5968.357585] LustreError: 197219:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5968.384839] LustreError: 197219:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5968.497815] Lustre: server umount lustre-MDT0000 complete [ 5972.065669] LustreError: 200029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754008 with bad export cookie 3781082383539976200 [ 5972.070222] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5972.078405] LustreError: 200029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5972.485231] Lustre: server umount lustre-MDT0001 complete [ 5984.358783] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5987.664044] Key type lgssc unregistered [ 5988.039446] LNet: 201085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5988.053770] LNetError: 201085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5989.092728] LNet: Removed LNI 192.168.201.126@tcp [ 5989.861051] Key type .llcrypt unregistered [ 5989.866060] Key type ._llcrypt unregistered [ 6010.508713] Key type ._llcrypt registered [ 6010.512072] Key type .llcrypt registered [ 6010.685702] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 6012.366534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6012.520186] alg: No test for adler32 (adler32-zlib) [ 6013.573729] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 6013.894856] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 6015.591543] Key type lgssc registered [ 6017.049242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6025.694350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6025.735316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.038611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6030.721311] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6037.480243] LustreError: 202347:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6037.488073] LustreError: 202347:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6040.879340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6041.284926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6047.866473] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6056.090948] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6061.785352] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6068.974310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6069.446759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6073.541639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6075.977577] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6084.503379] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6090.770512] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6100.538721] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6108.776852] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6113.049967] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6122.979261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6123.002349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6123.009226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6123.030472] Lustre: Skipped 1 previous similar message [ 6127.870686] Lustre: server umount lustre-OST0000 complete [ 6133.219464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6133.236704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6133.248360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6133.261247] Lustre: Skipped 2 previous similar messages [ 6133.282979] Lustre: Skipped 1 previous similar message [ 6138.338149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6138.354138] Lustre: Skipped 1 previous similar message [ 6138.547829] Lustre: server umount lustre-MDT0000 complete [ 6142.242056] LustreError: 205677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754178 with bad export cookie 16257707574752435545 [ 6142.245213] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6142.251418] LustreError: 205677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6142.559465] Lustre: server umount lustre-MDT0001 complete [ 6154.842801] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 6157.958517] Key type lgssc unregistered [ 6158.242546] LNet: 207030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6158.255355] LNetError: 207030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6158.278723] LNet: Removed LNI 192.168.201.126@tcp [ 6159.454284] Key type .llcrypt unregistered [ 6159.457484] Key type ._llcrypt unregistered [ 6185.204856] Key type ._llcrypt registered [ 6185.207739] Key type .llcrypt registered [ 6185.318400] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 6186.809935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6186.991595] alg: No test for adler32 (adler32-zlib) [ 6188.182756] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 6188.786619] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 6190.495294] Key type lgssc registered [ 6191.426821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6199.088680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6199.098374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6200.610978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6205.797295] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6210.019087] LustreError: 208465:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6210.035824] LustreError: 208465:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6215.145784] LustreError: 208466:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6215.387452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6215.854782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6220.937388] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6229.811682] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6234.840192] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6242.018246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6242.469914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6245.585394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6249.246350] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6258.845770] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6271.464610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6271.493203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6271.545148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6275.552696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6275.554206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6275.557333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6277.567261] Lustre: server umount lustre-OST0000 complete [ 6280.673651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6280.683633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6280.696853] Lustre: Skipped 1 previous similar message [ 6280.702269] Lustre: Skipped 1 previous similar message [ 6285.793081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6285.799774] Lustre: Skipped 2 previous similar messages [ 6286.934898] Lustre: server umount lustre-MDT0000 complete [ 6290.921717] LustreError: 208465:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6290.958714] LustreError: 208465:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6291.080495] LustreError: 208446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754327 with bad export cookie 10494880595816656126 [ 6291.100591] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6291.114111] LustreError: 208446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6291.615697] Lustre: server umount lustre-MDT0001 complete [ 6302.330624] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 6305.248718] Key type lgssc unregistered [ 6305.494872] LNet: 212147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6305.498869] LNetError: 212147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6306.571450] LNet: Removed LNI 192.168.201.126@tcp [ 6307.811087] Key type .llcrypt unregistered [ 6307.813363] Key type ._llcrypt unregistered [ 6332.066779] Key type ._llcrypt registered [ 6332.071437] Key type .llcrypt registered [ 6332.258502] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 6333.816874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6334.048394] alg: No test for adler32 (adler32-zlib) [ 6335.025453] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 6335.258870] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 6336.920811] Key type lgssc registered [ 6338.572095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6345.708217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6345.724889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.213349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6351.421933] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6357.477596] LustreError: 213416:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6357.495310] LustreError: 213416:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6359.662754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.227394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6364.847634] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6373.779430] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6381.115365] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6388.750221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6389.272798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6393.410255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6394.507283] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6401.543689] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6418.914925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6418.920835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6418.931700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6420.967786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6420.968352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6420.974623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6423.458787] Lustre: server umount lustre-OST0000 complete [ 6431.207255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6431.236591] Lustre: Skipped 1 previous similar message [ 6431.244410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6431.249440] Lustre: Skipped 2 previous similar messages [ 6433.127435] Lustre: server umount lustre-MDT0000 complete [ 6436.320614] LustreError: 215536:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6436.341425] LustreError: 215536:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6436.921980] LustreError: 213397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754473 with bad export cookie 2912800978487302798 [ 6436.924123] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6436.941099] LustreError: 213397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6437.374626] Lustre: server umount lustre-MDT0001 complete [ 6449.264413] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 6452.084942] Key type lgssc unregistered [ 6452.373485] LNet: 217094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6452.378217] LNetError: 217094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6452.396379] LNet: Removed LNI 192.168.201.126@tcp [ 6453.262865] Key type .llcrypt unregistered [ 6453.265470] Key type ._llcrypt unregistered [ 6471.923472] Key type ._llcrypt registered [ 6471.926280] Key type .llcrypt registered [ 6472.047530] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 6473.001301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6473.014544] alg: No test for adler32 (adler32-zlib) [ 6474.042329] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 6474.244887] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 6475.975233] Key type lgssc registered [ 6477.000932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6486.264890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6486.280992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6487.647134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6493.458283] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6497.775537] LustreError: 218356:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6497.792218] LustreError: 218356:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6502.111356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6502.484642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6507.253441] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6514.847709] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6519.915586] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6526.541431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6527.028565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6532.989355] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6533.122284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6540.199406] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6548.489710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6548.539218] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6548.732348] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6554.535620] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6561.934884] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6563.319827] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6575.248973] Lustre: Permanently deactivating lustre-OST0001 [ 6575.278295] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6581.295478] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6592.635183] Lustre: Permanently reactivating lustre-OST0001 [ 6592.652379] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6592.659737] Lustre: Skipped 2 previous similar messages [ 6598.194468] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6598.206186] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6598.211664] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6598.221962] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6598.252514] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6607.003128] Lustre: server umount lustre-OST0001 complete [ 6607.871691] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6607.881880] Lustre: Skipped 1 previous similar message [ 6607.888366] LustreError: 220179:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6607.913188] LustreError: 220179:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6612.961739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6612.976152] LustreError: 220180:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6612.980193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6612.984248] Lustre: Skipped 1 previous similar message [ 6613.032227] LustreError: 220180:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6618.082107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6618.087555] LustreError: 221835:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6618.095388] Lustre: Skipped 2 previous similar messages [ 6618.118815] LustreError: 221835:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6623.205679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6623.210976] LustreError: 221991:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6623.220907] Lustre: Skipped 1 previous similar message [ 6623.244377] LustreError: 221991:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6625.253309] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6625.440969] Lustre: server umount lustre-OST0000 complete [ 6630.373243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6630.377255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6630.398347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6635.486886] Lustre: server umount lustre-MDT0000 complete [ 6638.561936] LustreError: 218356:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6638.577093] LustreError: 218356:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6639.205793] LustreError: 218336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754675 with bad export cookie 2448701686828383454 [ 6639.212270] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6639.567712] Lustre: server umount lustre-MDT0001 complete [ 6649.953248] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 6652.495340] Key type lgssc unregistered [ 6652.835778] LNet: 223438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6652.852550] LNetError: 223438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6653.928982] LNet: Removed LNI 192.168.201.126@tcp [ 6654.932637] Key type .llcrypt unregistered [ 6654.934403] Key type ._llcrypt unregistered [ 6696.989519] Key type ._llcrypt registered [ 6696.991833] Key type .llcrypt registered [ 6697.114882] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 6698.387992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6698.414993] alg: No test for adler32 (adler32-zlib) [ 6699.630756] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 6700.104445] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 6701.816452] Key type lgssc registered [ 6702.940168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6711.522272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6711.548348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6712.891193] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6712.908838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6713.043607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6718.226205] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6727.578391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6727.696816] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6727.754416] Lustre: 225638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6728.046990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6733.355799] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6741.330729] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6747.556292] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6753.800697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6753.994439] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6754.201934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6759.818168] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6762.104057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6767.388222] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6769.800788] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6774.280146] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6783.378332] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6791.060959] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6801.045917] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6818.807695] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6818.818713] Lustre: Skipped 1 previous similar message [ 6851.519428] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6851.534524] Lustre: Skipped 3 previous similar messages [ 6922.220233] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6922.222458] Lustre: Skipped 9 previous similar messages [ 6941.152989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6941.157471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6941.169299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6941.673581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6941.682228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6941.686542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6946.788338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6946.795697] Lustre: Skipped 1 previous similar message [ 6947.173882] Lustre: server umount lustre-OST0000 complete [ 6951.904899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6951.920716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6951.927584] Lustre: Skipped 1 previous similar message [ 6957.026687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6965.215132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6965.548427] Lustre: server umount lustre-MDT0000 complete [ 6968.956347] LustreError: 225632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755005 with bad export cookie 16210688418857932833 [ 6968.958291] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6968.972272] LustreError: 225632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6969.522782] Lustre: server umount lustre-MDT0001 complete [ 6980.295347] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 6983.326484] Key type lgssc unregistered [ 6983.601801] LNet: 230458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6983.608844] LNetError: 230458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6984.679733] LNet: Removed LNI 192.168.201.126@tcp [ 6985.543411] Key type .llcrypt unregistered [ 6985.545418] Key type ._llcrypt unregistered [ 7009.238729] Key type ._llcrypt registered [ 7009.240764] Key type .llcrypt registered [ 7009.345782] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 7010.418689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7010.566659] alg: No test for adler32 (adler32-zlib) [ 7011.779723] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 7012.045583] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 7013.719538] Key type lgssc registered [ 7015.094088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7023.167712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7023.200470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.707857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7029.644555] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7034.855053] LustreError: 231868:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7034.885514] LustreError: 231868:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7038.185116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7038.574481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7043.155835] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7050.831065] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7056.929027] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7063.786789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7064.447510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7070.604187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7071.568444] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7080.055340] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7096.289425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7096.292767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7096.308499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7097.512914] Lustre: server umount lustre-OST0000 complete [ 7103.456497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7103.460525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7103.475996] Lustre: Skipped 1 previous similar message [ 7103.489740] Lustre: Skipped 1 previous similar message [ 7108.578141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7108.581479] Lustre: Skipped 1 previous similar message [ 7113.696895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7113.711050] Lustre: Skipped 1 previous similar message [ 7114.721319] Lustre: 231179:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788755134/real 1788755134] req@00000000a2b802c9 x1875645613249280/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788755150 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7114.723449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7114.761542] Lustre: 231179:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7115.231375] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7115.351400] Lustre: server umount lustre-MDT0000 complete [ 7118.883222] Lustre: 231180:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788755139/real 1788755139] req@000000004972f8e1 x1875645613250944/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788755155 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7119.178736] LustreError: 231848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755155 with bad export cookie 17602701223424418013 [ 7119.185394] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7119.191951] LustreError: 231848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7119.501908] Lustre: server umount lustre-MDT0001 complete [ 7131.118586] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 7134.135666] Key type lgssc unregistered [ 7134.420068] LNet: 235556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7134.429774] LNetError: 235556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7134.456415] LNet: Removed LNI 192.168.201.126@tcp [ 7135.400551] Key type .llcrypt unregistered [ 7135.402431] Key type ._llcrypt unregistered [ 7158.354917] Key type ._llcrypt registered [ 7158.358063] Key type .llcrypt registered [ 7158.484739] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 7159.625559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7159.805249] alg: No test for adler32 (adler32-zlib) [ 7160.927644] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 7161.186630] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 7162.863238] Key type lgssc registered [ 7164.087278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7433.912754] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 7437.167699] Key type lgssc unregistered [ 7437.459718] LNet: 237573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7437.463749] LNetError: 237573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7437.483346] LNet: Removed LNI 192.168.201.126@tcp [ 7438.401938] Key type .llcrypt unregistered [ 7438.404185] Key type ._llcrypt unregistered [ 7460.782348] Key type ._llcrypt registered [ 7460.784348] Key type .llcrypt registered [ 7460.873817] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 7478.287838] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 7479.566106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7479.592104] alg: No test for adler32 (adler32-zlib) [ 7480.791714] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 7481.128532] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 7482.807309] Key type lgssc registered [ 7484.074739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7508.525330] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7182 sec ========== 00:32:23 (1788755543) [ 7510.387652] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:32:25 (1788755545) === [ 7513.928395] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:32:29 (1788755549) === [ 7565.344746] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 7568.240144] Key type lgssc unregistered [ 7568.609057] LNet: 242807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7568.615417] LNetError: 242807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7568.632148] LNet: Removed LNI 192.168.201.126@tcp [ 7569.402713] Key type .llcrypt unregistered [ 7569.408347] Key type ._llcrypt unregistered