[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 484766300 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2837896K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003331] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009018] pid_max: default: 32768 minimum: 301 [ 0.010182] LSM: Security Framework initializing [ 0.011067] Yama: becoming mindful. [ 0.012050] SELinux: Initializing. [ 0.013095] *** VALIDATE selinux *** [ 0.022684] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027696] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028186] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029154] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031065] *** VALIDATE tmpfs *** [ 0.033042] *** VALIDATE proc *** [ 0.034306] *** VALIDATE cgroup *** [ 0.036008] *** VALIDATE cgroup2 *** [ 0.037313] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040040] Spectre V2 : User space: Vulnerable [ 0.041016] Speculative Store Bypass: Vulnerable [ 0.044005] debug: unmapping init [mem 0xffffffff9bc59000-0xffffffff9bc60fff] [ 0.046287] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047889] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048027] ... version: 2 [ 0.049014] ... bit width: 48 [ 0.050016] ... generic registers: 4 [ 0.051022] ... value mask: 0000ffffffffffff [ 0.052028] ... max period: 00007fffffffffff [ 0.053020] ... fixed-purpose events: 3 [ 0.054018] ... event mask: 000000070000000f [ 0.055468] rcu: Hierarchical SRCU implementation. [ 0.057736] smp: Bringing up secondary CPUs ... [ 0.058697] x86: Booting SMP configuration: [ 0.059033] .... node #0, CPUs: #1 #2 #3 [ 0.065771] smp: Brought up 1 node, 4 CPUs [ 0.067012] smpboot: Max logical packages: 1 [ 0.068015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.126128] node 0 deferred pages initialised in 56ms [ 0.129102] devtmpfs: initialized [ 0.130190] x86/mm: Memory block size: 128MB [ 0.135007] gcov: version magic: 0x41383552 [ 0.139623] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143207] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147357] pinctrl core: initialized pinctrl subsystem [ 0.150492] [ 0.152010] ************************************************************* [ 0.154015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157054] ** ** [ 0.162069] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166021] ** ** [ 0.169014] ** This means that this kernel is built to expose internal ** [ 0.173027] ** IOMMU data structures, which may compromise security on ** [ 0.177014] ** your system. ** [ 0.181015] ** ** [ 0.185017] ** If you see this message and you are not debugging the ** [ 0.187012] ** kernel, report this immediately to your vendor! ** [ 0.189013] ** ** [ 0.191011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194016] ************************************************************* [ 0.197890] NET: Registered protocol family 16 [ 0.199510] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.205064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208137] cpuidle: using governor menu [ 0.209716] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212478] PCI: Using configuration type 1 for base access [ 0.214128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.221116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223054] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.226085] cryptd: max_cpu_qlen set to 1000 [ 0.230212] ACPI: Added _OSI(Module Device) [ 0.231013] ACPI: Added _OSI(Processor Device) [ 0.231988] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.233014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237456] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.244447] ACPI: Interpreter enabled [ 0.245100] ACPI: PM: (supports S0 S3 S4 S5) [ 0.247013] ACPI: Using IOAPIC for interrupt routing [ 0.248142] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.251549] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265084] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.272107] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.277525] acpiphp: Slot [2] registered [ 0.279101] acpiphp: Slot [5] registered [ 0.280130] acpiphp: Slot [6] registered [ 0.281175] acpiphp: Slot [7] registered [ 0.282149] acpiphp: Slot [8] registered [ 0.283210] acpiphp: Slot [9] registered [ 0.285121] acpiphp: Slot [10] registered [ 0.286200] acpiphp: Slot [3] registered [ 0.288095] acpiphp: Slot [4] registered [ 0.289106] acpiphp: Slot [11] registered [ 0.290115] acpiphp: Slot [12] registered [ 0.291000] acpiphp: Slot [13] registered [ 0.291000] acpiphp: Slot [14] registered [ 0.294109] acpiphp: Slot [15] registered [ 0.295109] acpiphp: Slot [16] registered [ 0.296095] acpiphp: Slot [17] registered [ 0.297134] acpiphp: Slot [18] registered [ 0.298093] acpiphp: Slot [19] registered [ 0.300127] acpiphp: Slot [20] registered [ 0.301103] acpiphp: Slot [21] registered [ 0.302071] acpiphp: Slot [22] registered [ 0.303013] acpiphp: Slot [23] registered [ 0.303981] acpiphp: Slot [24] registered [ 0.304056] acpiphp: Slot [25] registered [ 0.305034] acpiphp: Slot [26] registered [ 0.307168] acpiphp: Slot [27] registered [ 0.308094] acpiphp: Slot [28] registered [ 0.310126] acpiphp: Slot [29] registered [ 0.311119] acpiphp: Slot [30] registered [ 0.313073] acpiphp: Slot [31] registered [ 0.314132] PCI host bridge to bus 0000:00 [ 0.316021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.318025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.320026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.323027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.325024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.328033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.330213] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334067] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.336103] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.346019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.351086] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.354020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.356033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.357017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.358699] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.361816] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.364048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.367796] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.373020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.386016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.391015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.397017] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.418023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.441022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.451089] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.467021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.476019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.505021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.517581] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.527015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.535025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.552025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.561978] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.571019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.583021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.604023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.618000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.630021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.645444] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.664022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.675749] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.688024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.696028] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.719019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.739902] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.742469] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.747432] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.751386] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.753301] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.758208] iommu: Default domain type: Passthrough [ 0.759575] SCSI subsystem initialized [ 0.760412] ACPI: bus type USB registered [ 0.763225] usbcore: registered new interface driver usbfs [ 0.767311] usbcore: registered new interface driver hub [ 0.769094] usbcore: registered new device driver usb [ 0.770000] pps_core: LinuxPPS API ver. 1 registered [ 0.772014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.777146] PTP clock support registered [ 0.781129] EDAC MC: Ver: 3.0.0 [ 0.782168] PCI: Using ACPI for IRQ routing [ 0.785134] NetLabel: Initializing [ 0.787021] NetLabel: domain hash size = 128 [ 0.788011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.790153] NetLabel: unlabeled traffic allowed by default [ 0.792456] vgaarb: loaded [ 0.794385] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.796019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.804000] clocksource: Switched to clocksource kvm-clock [ 0.930327] VFS: Disk quotas dquot_6.6.0 [ 0.932079] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.934776] *** VALIDATE ramfs *** [ 0.936244] *** VALIDATE hugetlbfs *** [ 0.937723] pnp: PnP ACPI init [ 0.939935] pnp: PnP ACPI: found 6 devices [ 0.959270] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.962967] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.966512] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.969996] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.973696] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.977139] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.981460] NET: Registered protocol family 2 [ 0.984834] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.991678] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.996169] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.003274] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.006922] TCP: Hash tables configured (established 65536 bind 65536) [ 1.009789] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.013015] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.015524] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.017869] NET: Registered protocol family 1 [ 1.020340] RPC: Registered named UNIX socket transport module. [ 1.022559] RPC: Registered udp transport module. [ 1.024268] RPC: Registered tcp transport module. [ 1.026088] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.028502] NET: Registered protocol family 44 [ 1.030357] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.032958] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.035181] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.037689] PCI: CLS 0 bytes, default 64 [ 1.039347] Unpacking initramfs... [ 2.417227] debug: unmapping init [mem 0xffff9f633cc54000-0xffff9f633ffbffff] [ 2.421279] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.423604] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.426481] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.864097] Initialise system trusted keyrings [ 2.865888] Key type blacklist registered [ 2.867694] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.876702] zbud: loaded [ 2.880295] *** VALIDATE nfs *** [ 2.881519] *** VALIDATE nfs4 *** [ 2.882552] pstore: using deflate compression [ 2.885188] Platform Keyring initialized [ 2.990824] NET: Registered protocol family 38 [ 2.992192] Key type asymmetric registered [ 2.993294] Asymmetric key parser 'x509' registered [ 2.994621] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.997588] io scheduler mq-deadline registered [ 2.998916] io scheduler kyber registered [ 3.000129] io scheduler bfq registered [ 3.001497] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.004607] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.007283] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.009171] ACPI: Power Button [PWRF] [ 3.015174] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.021823] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.033288] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.039774] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.055903] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.084810] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.111510] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.116684] Non-volatile memory driver v1.3 [ 3.118809] Linux agpgart interface v0.103 [ 3.151616] virtio_blk virtio1: [vda] 150056 512-byte logical blocks (76.8 MB/73.3 MiB) [ 3.154630] vda: detected capacity change from 0 to 76828672 [ 3.171590] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.174579] vdb: detected capacity change from 0 to 1073741824 [ 3.198635] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.202851] vdc: detected capacity change from 0 to 2621440000 [ 3.219677] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.221880] vdd: detected capacity change from 0 to 2621440000 [ 3.235554] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.238301] vde: detected capacity change from 0 to 4294967296 [ 3.254194] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.256160] vdf: detected capacity change from 0 to 4294967296 [ 3.267836] libphy: Fixed MDIO Bus: probed [ 3.280222] usbcore: registered new interface driver usbserial_generic [ 3.281803] usbserial: USB Serial support registered for generic [ 3.283895] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.288379] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.289501] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.291937] mousedev: PS/2 mouse device common for all mice [ 3.293452] rtc_cmos 00:05: RTC can wake from S4 [ 3.295458] rtc_cmos 00:05: registered as rtc0 [ 3.296427] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.298380] intel_pstate: CPU model not supported [ 3.300730] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.304313] hid: raw HID events driver (C) Jiri Kosina [ 3.304861] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.306289] usbcore: registered new interface driver usbhid [ 3.309395] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.310061] usbhid: USB HID core driver [ 3.313572] drop_monitor: Initializing network drop monitor service [ 3.315023] Initializing XFRM netlink socket [ 3.316268] NET: Registered protocol family 10 [ 3.318125] Segment Routing with IPv6 [ 3.318837] NET: Registered protocol family 17 [ 3.319937] mpls_gso: MPLS GSO support [ 3.323510] RAS: Correctable Errors collector initialized. [ 3.324770] AVX version of gcm_enc/dec engaged. [ 3.326046] AES CTR mode by8 optimization enabled [ 3.385798] sched_clock: Marking stable (3385778083, 0)->(4281473131, -895695048) [ 3.389706] registered taskstats version 1 [ 3.390771] Loading compiled-in X.509 certificates [ 3.391873] zswap: loaded using pool lzo/zbud [ 3.409487] Key type big_key registered [ 3.419982] Key type encrypted registered [ 3.420935] ima: No TPM chip found, activating TPM-bypass! [ 3.422236] ima: Allocated hash algorithm: sha1 [ 3.423331] ima: No architecture policies found [ 3.425488] evm: Initialising EVM extended attributes: [ 3.427379] evm: security.selinux [ 3.428638] evm: security.ima [ 3.429772] evm: security.capability [ 3.431059] evm: HMAC attrs: 0x1 [ 3.433718] rtc_cmos 00:05: setting system clock to 2026-09-07 01:23:33 UTC (1788744213) [ 3.439373] debug: unmapping init [mem 0xffffffff9cc03000-0xffffffff9cdfffff] [ 3.441274] debug: unmapping init [mem 0xffffffff9b982000-0xffffffff9bc58fff] [ 3.449135] Write protecting the kernel read-only data: 28672k [ 3.451467] debug: unmapping init [mem 0xffffffff9a003000-0xffffffff9a1fffff] [ 3.453434] debug: unmapping init [mem 0xffffffff9a914000-0xffffffff9a9fffff] [ 3.489829] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.498744] systemd[1]: Detected virtualization kvm. [ 3.500966] systemd[1]: Detected architecture x86-64. [ 3.502453] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.530475] systemd[1]: No hostname configured. [ 3.532293] systemd[1]: Set hostname to . [ 3.533674] random: systemd: uninitialized urandom read (16 bytes read) [ 3.535626] systemd[1]: Initializing machine ID from random generator. [ 3.579492] random: ln: uninitialized urandom read (6 bytes read) [ 3.654041] random: systemd: uninitialized urandom read (16 bytes read) [ 3.656919] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.663652] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.668121] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.186690] device-mapper: uevent: version 1.0.3 [ 4.188362] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.961498] virtio_net virtio0 ens2: renamed from eth0 [ 4.988823] random: fast init done [ 5.146253] scsi host0: ata_piix [ 5.248163] scsi host1: ata_piix [ 5.251047] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.255817] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.895037] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.936178] random: crng init done [ 9.937634] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.599479] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.886110] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.164034] SELinux: Disabled at runtime. [ 12.223199] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.232517] systemd[1]: Detected virtualization kvm. [ 12.234953] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.715924] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.719516] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.723825] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.729323] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.734474] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.742251] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.749496] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.845356] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.199985] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.578518] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.604918] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.707985] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.723164] EDAC sbridge: Ver: 1.1.2 [ 15.190604] Key type dns_resolver registered [ 15.517121] NFS: Registering the id_resolver key type [ 15.518748] Key type id_resolver registered [ 15.519782] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg141-server login: [ 68.400810] libcfs: loading out-of-tree module taints kernel. [ 68.486457] Key type ._llcrypt registered [ 68.490248] Key type .llcrypt registered [ 68.698764] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 91.638745] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 93.223846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 93.236565] alg: No test for adler32 (adler32-zlib) [ 94.696525] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 95.728535] hrtimer: interrupt took 1573155 ns [ 95.750221] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 97.525282] Key type lgssc registered [ 99.519107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.211817] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 166.221885] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 181.358461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 181.393050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.703162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.745458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.847297] Lustre: lustre-MDT0000: new disk, initializing [ 183.013493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.038857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 187.891829] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.304380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.430853] Lustre: 6535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.460797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 202.469719] Lustre: Skipped 1 previous similar message [ 202.571864] Lustre: lustre-MDT0001: new disk, initializing [ 202.630631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.649945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 202.671444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 208.059244] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.928493] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 222.079417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.357537] Lustre: lustre-OST0000: new disk, initializing [ 222.367857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 222.378602] Lustre: 8472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.455464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.186106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 224.200215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 224.302045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 228.864755] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.682284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 243.820794] Lustre: lustre-OST0001: new disk, initializing [ 243.824290] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 243.828912] Lustre: 9544:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 243.907798] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.882743] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.439966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 252.479946] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 252.601553] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 262.093414] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 269.307052] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 275.735492] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing check_logdir /tmp/testlogs/ [ 280.963855] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing yml_node [ 286.818711] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 289.621669] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 292.275617] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 293.929536] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 21:28:23 EDT 2026 [ 312.938792] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 314.713840] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 334.817603] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.818612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 334.831720] Lustre: Skipped 1 previous similar message [ 334.845169] Lustre: Skipped 3 previous similar messages [ 337.974260] Lustre: server umount lustre-MDT0000 complete [ 341.645589] LustreError: 6525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788744551 with bad export cookie 12870130990073054025 [ 341.646506] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 341.655470] LustreError: 6525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 345.056167] LustreError: 10378:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.057200] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.087876] LustreError: 10378:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 345.115169] Lustre: Skipped 2 previous similar messages [ 345.144586] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.080164] LustreError: 6542:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.113238] LustreError: 6542:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 348.452154] Lustre: server umount lustre-MDT0001 complete [ 352.720156] Lustre: server umount lustre-OST0000 complete [ 357.851722] Lustre: server umount lustre-OST0001 complete [ 367.743306] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 380.813225] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 415.464107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.686855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 415.711447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 415.772725] Lustre: lustre-MDT0000: new disk, initializing [ 415.861788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.878964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 421.068536] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 432.207738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.377488] Lustre: 15099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 432.450764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 432.459992] Lustre: Skipped 1 previous similar message [ 432.559799] Lustre: lustre-MDT0001: new disk, initializing [ 432.706247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 432.755159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 432.777962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 437.812186] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 448.893116] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 455.187437] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 462.174944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.522466] Lustre: lustre-OST0000: new disk, initializing [ 462.528552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 462.541599] Lustre: 17069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 462.661715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 463.707710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 463.727920] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 463.860871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 469.225188] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 479.342811] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 483.828890] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 484.102597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 488.691208] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 488.884368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 494.584525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 494.593608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.611438] Lustre: Skipped 1 previous similar message [ 494.618441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 494.622259] Lustre: Skipped 3 previous similar messages [ 499.685762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 499.694723] Lustre: Skipped 1 previous similar message [ 504.810092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 504.812428] Lustre: Skipped 1 previous similar message [ 505.311711] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 505.544893] Lustre: server umount lustre-OST0000 complete [ 509.409468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 509.430057] LustreError: Skipped 1 previous similar message [ 509.437106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.455845] Lustre: Skipped 1 previous similar message [ 515.042237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 515.052102] Lustre: Skipped 3 previous similar messages [ 515.749568] Lustre: server umount lustre-MDT0000 complete [ 519.315279] LustreError: 15091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788744729 with bad export cookie 12870130990073055558 [ 519.316768] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 519.323591] LustreError: 15091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 519.827343] Lustre: server umount lustre-MDT0001 complete [ 523.682664] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 21:32:13 (1788744733) [ 530.710048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.314783] LustreError: 19335:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.397671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 536.037897] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 536.546043] LustreError: 19336:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.671405] LustreError: 19335:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.292467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.610448] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 560.655701] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 568.656933] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 576.490745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 583.878577] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 592.944670] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 611.830275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 611.837962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.841805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 611.855191] Lustre: Skipped 2 previous similar messages [ 617.788922] Lustre: server umount lustre-OST0000 complete [ 622.050801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.074299] Lustre: Skipped 1 previous similar message [ 627.730226] Lustre: server umount lustre-MDT0000 complete [ 631.725555] LustreError: 19318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788744841 with bad export cookie 12870130990073056440 [ 631.743691] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 631.744303] LustreError: 19318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 632.097725] Lustre: server umount lustre-MDT0001 complete [ 643.621271] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 646.319428] Key type lgssc unregistered [ 646.561378] LNet: 22889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.566992] LNetError: 22889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 646.587528] LNet: Removed LNI 192.168.201.141@tcp [ 647.372164] Key type .llcrypt unregistered [ 647.375684] Key type ._llcrypt unregistered [ 671.093790] Key type ._llcrypt registered [ 671.096613] Key type .llcrypt registered [ 671.263318] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 672.349172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 672.577441] alg: No test for adler32 (adler32-zlib) [ 673.739678] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 674.002388] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 675.695263] Key type lgssc registered [ 676.802727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 684.662094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 684.683345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.227231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 690.795083] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 696.803076] LustreError: 24308:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 696.825357] LustreError: 24308:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 699.010294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.381871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 704.223730] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 712.238211] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 718.063915] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 724.614305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 724.952201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 731.125717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 731.247918] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 738.908161] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 755.683795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 755.695800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.715020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 756.707622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.714728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 756.734066] Lustre: Skipped 1 previous similar message [ 757.405431] Lustre: server umount lustre-OST0000 complete [ 761.826738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 761.829312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 761.844639] Lustre: Skipped 1 previous similar message [ 761.862182] Lustre: Skipped 1 previous similar message [ 766.944148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 766.964709] Lustre: Skipped 1 previous similar message [ 767.274890] Lustre: server umount lustre-MDT0000 complete [ 771.211926] LustreError: 24287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788744981 with bad export cookie 12480639843791990352 [ 771.212562] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 771.222690] LustreError: 24287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 771.818455] Lustre: server umount lustre-MDT0001 complete [ 784.146310] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 787.644158] Key type lgssc unregistered [ 788.114907] LNet: 28284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.124485] LNetError: 28284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.149194] LNet: Removed LNI 192.168.201.141@tcp [ 789.133173] Key type .llcrypt unregistered [ 789.134493] Key type ._llcrypt unregistered [ 815.096580] Key type ._llcrypt registered [ 815.098563] Key type .llcrypt registered [ 815.227771] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 816.181408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 816.254970] alg: No test for adler32 (adler32-zlib) [ 817.270296] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 817.532919] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 819.247288] Key type lgssc registered [ 820.612500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 829.517606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 829.531891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 830.906247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 835.357213] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 840.164074] LustreError: 29695:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 840.189418] LustreError: 29695:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 844.258426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.549466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 848.858843] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 855.824684] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 860.671764] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 875.801530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 876.113787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.164614] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 891.579603] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 900.712546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 910.305093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 910.317259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.343525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.328312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 911.336775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.345271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.392808] Lustre: Skipped 1 previous similar message [ 914.839282] Lustre: server umount lustre-OST0000 complete [ 921.569957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.601514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 921.620455] Lustre: Skipped 1 previous similar message [ 926.666356] Lustre: server umount lustre-MDT0000 complete [ 926.696040] LustreError: 31394:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.712729] LustreError: 31394:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 931.820183] LustreError: 29688:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 931.858097] LustreError: 29688:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 932.032666] LustreError: 29675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745142 with bad export cookie 8934929830293853430 [ 932.046365] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 932.578928] Lustre: server umount lustre-MDT0001 complete [ 945.463772] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 949.071923] Key type lgssc unregistered [ 949.465617] LNet: 33598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.484597] LNetError: 33598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.517189] LNet: Removed LNI 192.168.201.141@tcp [ 950.699055] Key type .llcrypt unregistered [ 950.700383] Key type ._llcrypt unregistered [ 981.542608] Key type ._llcrypt registered [ 981.545472] Key type .llcrypt registered [ 981.700951] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 983.545207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 983.765335] alg: No test for adler32 (adler32-zlib) [ 985.085581] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 985.445879] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 987.191297] Key type lgssc registered [ 988.455884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 996.378340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 996.396187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 997.931976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.656124] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1007.075267] LustreError: 35008:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1007.094866] LustreError: 35008:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1012.197779] LustreError: 35009:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.733234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.267300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1018.204681] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1027.496885] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1033.859447] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1041.036866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1041.444866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1042.490359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1048.831709] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1057.521278] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1068.006950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1068.024289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.068586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1069.025284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1069.032199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1069.037464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.073719] Lustre: Skipped 1 previous similar message [ 1071.843268] Lustre: server umount lustre-OST0000 complete [ 1079.271594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.285885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1079.293277] Lustre: Skipped 1 previous similar message [ 1079.312254] Lustre: Skipped 1 previous similar message [ 1081.933840] Lustre: server umount lustre-MDT0000 complete [ 1084.386838] LustreError: 35005:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.400904] LustreError: 35005:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1085.446724] LustreError: 34991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745295 with bad export cookie 10647821312281627796 [ 1085.457311] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1085.457462] LustreError: 34991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1085.741779] Lustre: server umount lustre-MDT0001 complete [ 1099.372834] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1102.217679] Key type lgssc unregistered [ 1102.666606] LNet: 38547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1102.677068] LNetError: 38547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1102.697767] LNet: Removed LNI 192.168.201.141@tcp [ 1103.815562] Key type .llcrypt unregistered [ 1103.826145] Key type ._llcrypt unregistered [ 1125.143484] Key type ._llcrypt registered [ 1125.145416] Key type .llcrypt registered [ 1125.250268] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1126.385218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1126.469038] alg: No test for adler32 (adler32-zlib) [ 1127.484343] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 1127.685097] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1129.361413] Key type lgssc registered [ 1130.663964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1137.875858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1137.889932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1139.421075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.653412] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1149.930401] LustreError: 39937:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.960366] LustreError: 39937:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1155.042219] LustreError: 39938:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1155.175687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1155.624977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1159.751148] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1168.127748] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1172.873853] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1180.398033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1180.688085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1184.840390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1188.272362] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1196.215772] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1208.533863] Lustre: lustre-OST0000: Not available for connect from 192.168.201.41@tcp (stopping) [ 1210.340650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1210.343498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1210.362127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1211.885410] Lustre: server umount lustre-OST0000 complete [ 1232.354565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1232.366067] LustreError: Skipped 1 previous similar message [ 1232.368195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.384340] Lustre: Skipped 1 previous similar message [ 1232.411035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1232.423746] Lustre: Skipped 1 previous similar message [ 1235.941230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.947038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1235.972583] Lustre: Skipped 1 previous similar message [ 1238.227665] Lustre: server umount lustre-MDT0000 complete [ 1241.056020] LustreError: 39938:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.084665] LustreError: 39938:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1242.143793] LustreError: 39917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745452 with bad export cookie 4172869764605523983 [ 1242.148922] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1242.161800] LustreError: 39917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1248.756089] Lustre: server umount lustre-MDT0001 complete [ 1259.226582] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1261.980834] Key type lgssc unregistered [ 1262.321589] LNet: 43646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.332937] LNetError: 43646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.363467] LNet: Removed LNI 192.168.201.141@tcp [ 1263.221129] Key type .llcrypt unregistered [ 1263.223260] Key type ._llcrypt unregistered [ 1285.245436] Key type ._llcrypt registered [ 1285.247477] Key type .llcrypt registered [ 1285.389068] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1286.531761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1286.687643] alg: No test for adler32 (adler32-zlib) [ 1287.676842] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 1287.986871] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1289.631668] Key type lgssc registered [ 1290.706289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1298.130632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1298.161813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1299.586774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1303.814817] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1309.158429] LustreError: 45047:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1309.177989] LustreError: 45047:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1312.415049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1312.914507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1317.281488] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1325.891901] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1334.031757] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1342.281369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1342.660781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.735602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1349.103557] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1355.709131] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1362.401163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.407713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1362.426184] Lustre: Skipped 2 previous similar messages [ 1362.433567] Lustre: Skipped 2 previous similar messages [ 1363.723594] Lustre: server umount lustre-MDT0000 complete [ 1367.389185] LustreError: 45030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745577 with bad export cookie 13292510152005047507 [ 1367.396843] LustreError: 45030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1367.397773] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1367.520940] LustreError: 46648:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.528520] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.556566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1367.560108] Lustre: Skipped 1 previous similar message [ 1372.641688] LustreError: 45044:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.641961] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1372.661619] LustreError: 45044:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1372.672422] Lustre: Skipped 1 previous similar message [ 1373.900661] Lustre: server umount lustre-MDT0001 complete [ 1404.383959] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1404.715170] Lustre: server umount lustre-OST0000 complete [ 1422.580818] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1425.536671] Key type lgssc unregistered [ 1425.877429] LNet: 48906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1425.882498] LNetError: 48906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1425.900470] LNet: Removed LNI 192.168.201.141@tcp [ 1427.242930] Key type .llcrypt unregistered [ 1427.248447] Key type ._llcrypt unregistered [ 1455.721474] Key type ._llcrypt registered [ 1455.729109] Key type .llcrypt registered [ 1456.039080] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1457.382114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1457.512617] alg: No test for adler32 (adler32-zlib) [ 1458.627523] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 1458.804590] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1460.423476] Key type lgssc registered [ 1461.238769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1468.965103] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1469.009482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1484.575638] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1489.695356] LustreError: 50294:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 1516.383949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.120133] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1533.396749] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1699.036768] Lustre: server umount lustre-OST0000 complete [ 1712.160765] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1714.292849] Key type lgssc unregistered [ 1714.508685] LNet: 51970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.514878] LNetError: 51970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1714.533342] LNet: Removed LNI 192.168.201.141@tcp [ 1715.198148] Key type .llcrypt unregistered [ 1715.199747] Key type ._llcrypt unregistered [ 1735.962825] Key type ._llcrypt registered [ 1735.965390] Key type .llcrypt registered [ 1736.073923] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1736.956465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1737.005762] alg: No test for adler32 (adler32-zlib) [ 1738.073394] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 1738.284158] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1739.967265] Key type lgssc registered [ 1741.203965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1748.341602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1748.353794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1749.794495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.450824] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1759.208151] LustreError: 53381:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1759.221130] LustreError: 53381:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1763.571295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1763.850853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1768.363783] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1775.085469] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1780.462221] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1786.999629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1787.324523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.726160] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1794.480970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1799.345175] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1803.747174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1803.755922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.768384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1804.780124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1804.788864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1804.806552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.902088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1809.911748] Lustre: Skipped 1 previous similar message [ 1815.014339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1815.030641] Lustre: Skipped 1 previous similar message [ 1816.034763] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1816.212779] Lustre: server umount lustre-OST0000 complete [ 1820.129548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1820.130699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1820.141289] Lustre: Skipped 1 previous similar message [ 1820.156069] Lustre: Skipped 2 previous similar messages [ 1826.241532] Lustre: server umount lustre-MDT0000 complete [ 1829.404412] LustreError: 53362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746039 with bad export cookie 13056128866849931245 [ 1829.416389] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1829.703452] Lustre: server umount lustre-MDT0001 complete [ 1840.845429] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1843.316971] Key type lgssc unregistered [ 1843.581983] LNet: 56910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.594369] LNetError: 56910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.612136] LNet: Removed LNI 192.168.201.141@tcp [ 1844.376994] Key type .llcrypt unregistered [ 1844.379428] Key type ._llcrypt unregistered [ 1865.277677] Key type ._llcrypt registered [ 1865.279805] Key type .llcrypt registered [ 1865.368441] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1865.926824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1866.051893] alg: No test for adler32 (adler32-zlib) [ 1867.094936] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 1867.395385] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1869.015273] Key type lgssc registered [ 1870.255730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1878.310410] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1878.343830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1894.048143] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1899.167623] LustreError: 58294:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 1925.855339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.969897] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1940.294523] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1946.496286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.978704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.043037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1950.920794] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1952.229449] LustreError: 58319:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1952.249305] LustreError: 58319:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1957.353516] LustreError: 58320:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.604652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.961101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1964.580602] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1973.233462] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1978.944417] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1981.414771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.425722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1981.439992] Lustre: Skipped 1 previous similar message [ 1986.530444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1986.539842] Lustre: Skipped 1 previous similar message [ 1991.679590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1991.701291] Lustre: Skipped 2 previous similar messages [ 1995.231349] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1995.456444] Lustre: server umount lustre-OST0000 complete [ 2017.251509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.267724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2017.276240] Lustre: Skipped 1 previous similar message [ 2021.346934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.356110] Lustre: Skipped 1 previous similar message [ 2023.060312] Lustre: server umount lustre-MDT0000 complete [ 2026.465206] LustreError: 60919:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.493183] LustreError: 60919:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2027.055195] LustreError: 58301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746237 with bad export cookie 11031300455321441251 [ 2027.089020] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2033.510654] Lustre: server umount lustre-MDT0001 complete [ 2044.860548] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2047.557536] Key type lgssc unregistered [ 2047.938574] LNet: 62039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2047.946373] LNetError: 62039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2047.976336] LNet: Removed LNI 192.168.201.141@tcp [ 2049.378851] Key type .llcrypt unregistered [ 2049.384458] Key type ._llcrypt unregistered [ 2075.451918] Key type ._llcrypt registered [ 2075.454778] Key type .llcrypt registered [ 2075.580497] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2076.473904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2076.616408] alg: No test for adler32 (adler32-zlib) [ 2077.778531] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 2077.937460] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2079.575399] Key type lgssc registered [ 2080.389327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2088.190417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2088.216208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.869201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.456282] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2099.173533] LustreError: 63457:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.238828] LustreError: 63457:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2104.291502] LustreError: 63458:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.863550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.411652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2115.505584] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2129.290929] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2136.568228] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2144.944216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2145.270718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.037728] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2159.228685] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2159.603860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2179.120503] Lustre: server umount lustre-OST0000 complete [ 2185.187230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2185.188504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2185.189634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2185.211061] Lustre: Skipped 1 previous similar message [ 2190.307764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2190.320733] Lustre: Skipped 2 previous similar messages [ 2193.650073] Lustre: server umount lustre-MDT0000 complete [ 2195.447669] Lustre: 62769:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788746389/real 1788746389] req@00000000d6a1a07e x1875636426669056/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788746405 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2195.467021] LustreError: 63458:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.507599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2195.572467] LustreError: 63458:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2196.512528] Lustre: 62771:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788746390/real 1788746390] req@000000009b8bfef6 x1875636426669824/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788746406 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2198.765996] LustreError: 63439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746408 with bad export cookie 306320577106822119 [ 2198.803392] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2199.840546] Lustre: server umount lustre-MDT0001 complete [ 2213.298618] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2216.735841] Key type lgssc unregistered [ 2216.996708] LNet: 67016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.004680] LNetError: 67016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.026246] LNet: Removed LNI 192.168.201.141@tcp [ 2218.189161] Key type .llcrypt unregistered [ 2218.190800] Key type ._llcrypt unregistered [ 2250.382189] Key type ._llcrypt registered [ 2250.383878] Key type .llcrypt registered [ 2250.601751] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2252.692552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2252.702551] alg: No test for adler32 (adler32-zlib) [ 2253.975436] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 2254.296533] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2256.031835] Key type lgssc registered [ 2257.403749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2267.173167] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 22:01:16 (1788746476) [ 2274.832493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2274.875165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.460821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2281.935589] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2286.067095] LustreError: 68737:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2286.086653] LustreError: 68737:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2291.170786] LustreError: 68738:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.052912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2292.612601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2297.413106] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2305.007036] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2309.936780] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2317.741840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2318.230600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2321.403114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2325.877333] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2334.035326] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2340.075818] Lustre: Failing over lustre-MDT0000 [ 2340.305397] Lustre: server umount lustre-MDT0000 complete [ 2341.064623] LustreError: 68732:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.871177] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.886170] Lustre: Skipped 1 previous similar message [ 2346.168737] LustreError: 68734:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.185385] LustreError: 68734:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 2349.102635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.225450] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2349.360943] Lustre: *** cfs_fail_loc=135, val=0*** [ 2349.505962] LustreError: 71823:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2349.519073] LustreError: 71823:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.141@tcp: cfg command failed: rc = -2 [ 2349.528617] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2349.528617] [ 2349.542450] LustreError: MGC192.168.201.141@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2349.555515] LustreError: 71811:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2349.564806] LustreError: 71811:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2349.582781] LustreError: 71811:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2349.601610] LustreError: 71811:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2349.644266] Lustre: server umount lustre-MDT0000 complete [ 2349.649855] LustreError: 71811:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2351.282547] LustreError: 68732:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.304448] LustreError: 68732:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 2356.790199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.010011] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2357.257975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2357.304256] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2357.467327] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2362.055678] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2362.446029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2362.500241] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2362.552591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2370.528473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2370.542518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.550877] Lustre: Skipped 1 previous similar message [ 2370.554718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2372.588099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2372.593024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.552160] Lustre: server umount lustre-OST0000 complete [ 2382.830565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.853480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2382.858530] Lustre: Skipped 1 previous similar message [ 2382.884106] Lustre: Skipped 2 previous similar messages [ 2386.882435] Lustre: server umount lustre-MDT0000 complete [ 2387.939398] LustreError: 70580:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.955953] LustreError: 70580:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 2390.513977] LustreError: 68720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746600 with bad export cookie 2020581865620511067 [ 2390.527359] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2390.531318] LustreError: 68720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2390.810160] Lustre: server umount lustre-MDT0001 complete [ 2401.429539] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2404.235770] Key type lgssc unregistered [ 2404.513518] LNet: 73602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2404.519988] LNetError: 73602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2404.544224] LNet: Removed LNI 192.168.201.141@tcp [ 2405.378446] Key type .llcrypt unregistered [ 2405.380134] Key type ._llcrypt unregistered [ 2428.178761] Key type ._llcrypt registered [ 2428.189677] Key type .llcrypt registered [ 2428.336621] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2429.606281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2429.629486] alg: No test for adler32 (adler32-zlib) [ 2430.724607] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 2430.918903] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2432.607666] Key type lgssc registered [ 2433.680943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2442.170734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2442.199478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2443.714803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.878407] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2452.962834] LustreError: 75011:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.977360] LustreError: 75011:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2456.427750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.020210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2462.256883] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2469.743941] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2475.253931] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2483.637955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2483.942180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2488.059111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2490.380832] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2498.986627] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2508.299473] Lustre: Failing over lustre-MDT0000 [ 2508.700622] Lustre: server umount lustre-MDT0000 complete [ 2511.052593] LustreError: 75008:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.843308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.861264] Lustre: Skipped 1 previous similar message [ 2516.147415] LustreError: 75989:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.179695] LustreError: 75989:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 2518.864545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.026942] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2519.163305] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2519.274732] LustreError: 78098:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2519.282000] LustreError: 78098:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.141@tcp: cfg command failed: rc = -9 [ 2519.292216] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2519.292216] [ 2519.299965] LustreError: MGC192.168.201.141@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2519.308303] LustreError: 78086:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2519.314695] LustreError: 78086:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2519.322545] LustreError: 78086:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2519.327899] LustreError: 78086:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2519.362464] Lustre: server umount lustre-MDT0000 complete [ 2519.364739] LustreError: 78086:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2521.271222] LustreError: 75008:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.293267] LustreError: 75008:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 2526.119750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2526.210350] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2526.220583] LustreError: 78446:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.201.141@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000779d24bd) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2526.238083] LustreError: 75002:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2526.246816] LustreError: 75989:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.257274] LustreError: 75989:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 2526.390482] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.41@tcp (not set up) [ 2526.576628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.633387] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2530.966653] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2531.503729] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2531.818525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2531.868935] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2531.911158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2542.054406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.070899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2542.073285] Lustre: Skipped 2 previous similar messages [ 2542.081892] Lustre: Skipped 1 previous similar message [ 2547.170121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2547.181186] Lustre: Skipped 1 previous similar message [ 2552.287263] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2552.295565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2552.313334] Lustre: Skipped 1 previous similar message [ 2552.472594] Lustre: server umount lustre-OST0000 complete [ 2557.408646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.417710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2557.421626] Lustre: Skipped 1 previous similar message [ 2562.529454] LustreError: 75008:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.548600] LustreError: 75008:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2562.591089] Lustre: server umount lustre-MDT0000 complete [ 2565.856891] LustreError: 74991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746775 with bad export cookie 12870550189561197353 [ 2565.858950] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2565.864469] LustreError: 74991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2566.160702] Lustre: server umount lustre-MDT0001 complete [ 2575.896330] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2578.387416] Key type lgssc unregistered [ 2578.682643] LNet: 79883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2578.685525] LNetError: 79883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2578.696356] LNet: Removed LNI 192.168.201.141@tcp [ 2579.534110] Key type .llcrypt unregistered [ 2579.542505] Key type ._llcrypt unregistered [ 2591.843389] Key type ._llcrypt registered [ 2591.845354] Key type .llcrypt registered [ 2591.975564] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2592.311919] Key type .llcrypt unregistered [ 2592.320611] Key type ._llcrypt unregistered [ 2614.786762] Key type ._llcrypt registered [ 2614.791105] Key type .llcrypt registered [ 2614.930538] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2615.779680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2615.847224] alg: No test for adler32 (adler32-zlib) [ 2616.945507] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 2617.139502] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2618.792283] Key type lgssc registered [ 2620.098528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2628.946765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2628.972164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2630.779535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2636.125929] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2640.369358] LustreError: 81778:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.393469] LustreError: 81778:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2644.534278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.943439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2649.405451] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2658.633506] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2665.292594] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2671.645944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2671.866115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2675.957839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2676.954336] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2683.099679] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2694.630179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.639434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2694.642266] Lustre: Skipped 1 previous similar message [ 2694.657639] Lustre: Skipped 1 previous similar message [ 2699.743870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2699.758780] Lustre: Skipped 1 previous similar message [ 2700.285667] Lustre: server umount lustre-OST0000 complete [ 2704.866509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.872139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2704.886482] Lustre: Skipped 1 previous similar message [ 2709.849031] Lustre: server umount lustre-MDT0000 complete [ 2709.986746] LustreError: 81778:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.022372] LustreError: 81778:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2712.996808] LustreError: 81759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746923 with bad export cookie 2743875566283779869 [ 2713.001783] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2713.005483] LustreError: 81759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2713.277537] Lustre: server umount lustre-MDT0001 complete [ 2722.843573] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2725.132941] Key type lgssc unregistered [ 2725.388257] LNet: 85308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2725.395340] LNetError: 85308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2725.407963] LNet: Removed LNI 192.168.201.141@tcp [ 2725.950295] Key type .llcrypt unregistered [ 2725.951823] Key type ._llcrypt unregistered [ 2745.712293] Key type ._llcrypt registered [ 2745.714325] Key type .llcrypt registered [ 2745.856680] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2746.839114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2746.865787] alg: No test for adler32 (adler32-zlib) [ 2747.946121] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 2748.178312] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2749.855191] Key type lgssc registered [ 2750.856346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2757.693369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2757.708595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.073446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2762.932989] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2769.395662] LustreError: 86696:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.429359] LustreError: 86696:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2770.889783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.233744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2775.456575] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2782.682347] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2788.210663] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2794.694913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2794.997232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2800.755914] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2804.223176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2807.191772] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2814.436066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2814.439421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.453130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2814.467929] Lustre: Skipped 1 previous similar message [ 2814.481724] Lustre: Skipped 1 previous similar message [ 2819.421346] Lustre: server umount lustre-OST0000 complete [ 2822.624686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2822.637177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.675980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.675066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2824.679154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2828.619805] Lustre: server umount lustre-MDT0000 complete [ 2829.793069] LustreError: 87742:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.805644] LustreError: 87742:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2831.519498] LustreError: 86677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747041 with bad export cookie 4408258702332777638 [ 2831.523808] LustreError: 86677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2831.525507] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2831.835796] Lustre: server umount lustre-MDT0001 complete [ 2840.112624] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2842.604987] Key type lgssc unregistered [ 2843.011903] LNet: 90227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2843.017896] LNetError: 90227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2843.030851] LNet: Removed LNI 192.168.201.141@tcp [ 2843.683815] Key type .llcrypt unregistered [ 2843.685493] Key type ._llcrypt unregistered [ 2864.459302] Key type ._llcrypt registered [ 2864.461724] Key type .llcrypt registered [ 2864.560249] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2865.613744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2865.633441] alg: No test for adler32 (adler32-zlib) [ 2866.574106] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 2866.773178] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2868.495274] Key type lgssc registered [ 2869.511949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2876.105875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2876.138892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.608296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2880.944585] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2883.049300] LustreError: 91617:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.068042] LustreError: 91617:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2887.153506] LustreError: 91618:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.860640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.141292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2891.378907] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2897.049942] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2902.342981] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2907.577749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2907.917097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2913.464320] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2919.562525] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2922.292167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2932.706806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.716907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2932.720139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2937.825952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2937.827902] Lustre: Skipped 1 previous similar message [ 2938.975946] Lustre: server umount lustre-OST0000 complete [ 2942.946086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.949207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.969422] Lustre: Skipped 2 previous similar messages [ 2942.984290] Lustre: Skipped 2 previous similar messages [ 2948.065554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2948.068532] Lustre: Skipped 1 previous similar message [ 2953.185447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2953.192550] Lustre: Skipped 1 previous similar message [ 2956.255869] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2956.388895] Lustre: server umount lustre-MDT0000 complete [ 2958.307796] LustreError: 92345:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.334277] LustreError: 92345:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2959.402869] LustreError: 91599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747169 with bad export cookie 17919262807553279126 [ 2959.413416] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2959.415029] LustreError: 91599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2959.650151] Lustre: server umount lustre-MDT0001 complete [ 2968.544842] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2971.017388] Key type lgssc unregistered [ 2971.252128] LNet: 95174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2971.260687] LNetError: 95174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2971.280529] LNet: Removed LNI 192.168.201.141@tcp [ 2971.970548] Key type .llcrypt unregistered [ 2971.972546] Key type ._llcrypt unregistered [ 2990.993305] Key type ._llcrypt registered [ 2990.994989] Key type .llcrypt registered [ 2991.079632] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2992.556856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2992.618080] alg: No test for adler32 (adler32-zlib) [ 2993.627315] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 2993.863967] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2995.559301] Key type lgssc registered [ 2996.841487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3009.538989] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3009.565087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.183824] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3030.304578] LustreError: 96537:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 3057.055228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3064.616894] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3073.505506] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3079.690906] Lustre: server umount lustre-OST0000 complete [ 3088.731148] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 22:14:57 (1788747297) [ 3095.340550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.958759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3101.153730] LustreError: 98177:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3101.165795] LustreError: 98177:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3101.754541] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3105.259581] LustreError: 98178:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.374566] LustreError: 98177:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.606702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3111.142288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3115.270092] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3123.789085] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3130.633589] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3139.446052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3139.831657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.877574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3148.653641] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3157.567459] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3171.814171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.819814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3171.820600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3171.829950] Lustre: Skipped 1 previous similar message [ 3171.861341] LustreError: Skipped 1 previous similar message [ 3176.326955] Lustre: server umount lustre-OST0000 complete [ 3182.065727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3182.079686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3182.087342] Lustre: Skipped 1 previous similar message [ 3182.113964] Lustre: Skipped 2 previous similar messages [ 3186.801780] Lustre: server umount lustre-MDT0000 complete [ 3187.169595] LustreError: 98177:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3187.193372] LustreError: 98177:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3191.330556] LustreError: 98157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747401 with bad export cookie 12928693266659601383 [ 3191.351366] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3191.790243] Lustre: server umount lustre-MDT0001 complete [ 3204.155116] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 3207.879832] Key type lgssc unregistered [ 3208.253745] LNet: 101871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3208.262856] LNetError: 101871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3208.283731] LNet: Removed LNI 192.168.201.141@tcp [ 3209.294729] Key type .llcrypt unregistered [ 3209.296946] Key type ._llcrypt unregistered [ 3254.757163] Key type ._llcrypt registered [ 3254.759193] Key type .llcrypt registered [ 3254.852502] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3256.035721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3256.145468] alg: No test for adler32 (adler32-zlib) [ 3257.234115] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 3257.447819] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 3259.103229] Key type lgssc registered [ 3260.730730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3271.239283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3271.261593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3272.925610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3278.284588] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3282.401895] LustreError: 104052:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3282.420983] LustreError: 104052:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3287.533709] LustreError: 104053:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.566178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3287.969155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3292.531775] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3300.802416] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3307.319356] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3314.838953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3315.090451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3317.193313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3322.562736] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3330.820368] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3342.824113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.830633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3342.836991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3342.851980] Lustre: Skipped 1 previous similar message [ 3342.889508] Lustre: Skipped 1 previous similar message [ 3347.417311] Lustre: server umount lustre-OST0000 complete [ 3353.059052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.063703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3353.081497] Lustre: Skipped 1 previous similar message [ 3353.095874] Lustre: Skipped 1 previous similar message [ 3357.941136] Lustre: server umount lustre-MDT0000 complete [ 3358.181671] LustreError: 104053:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3358.221091] LustreError: 104053:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3362.272256] LustreError: 104035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747572 with bad export cookie 12547710669566742532 [ 3362.287565] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3362.293480] LustreError: 104035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3362.980324] Lustre: server umount lustre-MDT0001 complete [ 3376.981861] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 3380.109185] Key type lgssc unregistered [ 3380.520525] LNet: 107586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3380.538847] LNetError: 107586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3380.563897] LNet: Removed LNI 192.168.201.141@tcp [ 3381.721772] Key type .llcrypt unregistered [ 3381.728753] Key type ._llcrypt unregistered [ 3402.018848] Key type ._llcrypt registered [ 3402.023224] Key type .llcrypt registered [ 3402.193939] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3403.593615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3403.670248] alg: No test for adler32 (adler32-zlib) [ 3404.728532] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 3405.150596] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 3406.879906] Key type lgssc registered [ 3408.290042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3417.110148] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3417.132680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3432.799603] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3437.919624] LustreError: 108833:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 3464.609311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3471.801963] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3479.286586] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3486.983059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.249668] LustreError: MGC192.168.201.141@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3487.260430] LustreError: 109856:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3487.266437] LustreError: 109856:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3487.271713] LustreError: 109856:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3487.381448] Lustre: server umount lustre-MDT0000 complete [ 3487.385307] LustreError: 109856:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3505.508697] Lustre: server umount lustre-OST0000 complete [ 3516.029927] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 3529.290945] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3581.291216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.455818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3581.529178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3581.655561] Lustre: lustre-MDT0000: new disk, initializing [ 3581.797610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.817367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3586.696862] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3599.599130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3599.760863] Lustre: 112653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3599.814969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3599.819934] Lustre: Skipped 1 previous similar message [ 3599.892492] Lustre: lustre-MDT0001: new disk, initializing [ 3599.992978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3600.059020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3600.077602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3605.092363] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3619.033272] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3627.823287] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3634.909901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3635.267525] Lustre: lustre-OST0000: new disk, initializing [ 3635.270052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3635.278496] Lustre: 114622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3635.379966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3641.533325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3641.555693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3641.801190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3641.981532] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3652.949910] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3658.081532] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3658.377959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3664.169576] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3664.447408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3667.425821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3667.434981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3667.439643] LustreError: Skipped 1 previous similar message [ 3667.441810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3667.467319] Lustre: Skipped 1 previous similar message [ 3672.550851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3672.555992] Lustre: Skipped 1 previous similar message [ 3677.667693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3677.676213] Lustre: Skipped 2 previous similar messages [ 3681.247340] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3681.446781] Lustre: server umount lustre-OST0000 complete [ 3686.880924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3686.892070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3686.922047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3687.917866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3692.913273] Lustre: server umount lustre-MDT0000 complete [ 3697.909907] LustreError: 114621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747907 with bad export cookie 8653508672615180673 [ 3697.912026] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3697.920524] LustreError: 114621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3698.382420] Lustre: server umount lustre-MDT0001 complete [ 3708.096394] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 22:25:17 (1788747917) [ 3710.941838] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3727.949759] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 3738.179706] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3775.740198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3776.125464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3776.181714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3776.296263] Lustre: lustre-MDT0000: new disk, initializing [ 3776.437836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3776.463443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3781.581403] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3793.815575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3793.899473] Lustre: 119525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3793.950759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3793.954430] Lustre: Skipped 1 previous similar message [ 3794.023654] Lustre: lustre-MDT0001: new disk, initializing [ 3794.123202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3794.157071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3794.163502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3799.475817] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3809.580108] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3814.988795] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3823.342904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3823.810798] Lustre: lustre-OST0000: new disk, initializing [ 3823.823898] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3823.840975] Lustre: 121495:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3824.036234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.909800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3829.935315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3830.056051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3833.359573] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3844.629835] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3849.820651] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3850.180673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3855.780995] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3856.079976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3860.961396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3860.965553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.968734] LustreError: Skipped 1 previous similar message [ 3860.985138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3860.989958] Lustre: Skipped 3 previous similar messages [ 3871.212925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3871.222668] Lustre: Skipped 3 previous similar messages [ 3873.247464] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3873.439773] Lustre: server umount lustre-OST0000 complete [ 3881.447139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3881.448194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3881.456634] Lustre: Skipped 2 previous similar messages [ 3883.648028] Lustre: server umount lustre-MDT0000 complete [ 3886.562103] LustreError: 119532:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3886.591499] LustreError: 119532:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 3888.069987] LustreError: 119517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748098 with bad export cookie 8653508672615181555 [ 3888.080076] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3888.083849] LustreError: 119517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3888.680578] Lustre: server umount lustre-MDT0001 complete [ 3897.058899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3897.628879] LustreError: 123715:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3897.785959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3902.946054] LustreError: 123716:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.165790] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3907.044885] LustreError: 123715:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3912.163550] LustreError: 123716:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3913.056635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3920.864878] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3930.993652] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3938.154383] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3947.430329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.286480] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3963.705951] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3975.240658] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3982.836381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.844439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3982.844727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3982.861429] Lustre: Skipped 1 previous similar message [ 3982.885339] Lustre: Skipped 3 previous similar messages [ 3985.515649] Lustre: server umount lustre-OST0000 complete [ 4004.320062] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4004.731443] Lustre: server umount lustre-MDT0000 complete [ 4008.416492] LustreError: 123712:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4008.429030] LustreError: 123712:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4008.736063] LustreError: 123697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748218 with bad export cookie 8653508672615182437 [ 4008.742383] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4008.754300] LustreError: 123697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4009.313305] Lustre: server umount lustre-MDT0001 complete [ 4023.756947] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4028.816638] Key type lgssc unregistered [ 4029.357241] LNet: 127347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4029.380783] LNetError: 127347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4029.428521] LNet: Removed LNI 192.168.201.141@tcp [ 4031.376449] Key type .llcrypt unregistered [ 4031.386800] Key type ._llcrypt unregistered [ 4057.042410] Key type ._llcrypt registered [ 4057.049700] Key type .llcrypt registered [ 4057.281671] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 4076.733465] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4078.178387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4078.212674] alg: No test for adler32 (adler32-zlib) [ 4079.225693] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 4079.426802] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4081.191219] Key type lgssc registered [ 4082.771218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4123.268032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4123.287995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4124.567958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4124.597756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4124.728463] Lustre: lustre-MDT0000: new disk, initializing [ 4124.878402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4124.923537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4129.863072] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4140.472049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4140.665781] Lustre: 130503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4140.801824] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4140.807654] Lustre: Skipped 1 previous similar message [ 4140.920314] Lustre: lustre-MDT0001: new disk, initializing [ 4141.020544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4141.091484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4141.111071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4146.357844] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4155.433833] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4161.261652] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4167.698898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4167.923800] Lustre: lustre-OST0000: new disk, initializing [ 4167.927161] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4167.932825] Lustre: 132472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4167.998179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4174.959796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4174.976930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4175.127645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4175.290584] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4184.617867] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4189.147499] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4189.436460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4194.731794] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4195.026741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4200.932979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.942727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4200.946270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4200.948763] Lustre: Skipped 1 previous similar message [ 4200.959306] LustreError: Skipped 1 previous similar message [ 4206.055032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4206.061129] Lustre: Skipped 2 previous similar messages [ 4211.176237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4211.181137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 4211.193481] Lustre: Skipped 1 previous similar message [ 4211.439931] Lustre: server umount lustre-OST0000 complete [ 4216.290153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4216.302399] Lustre: Skipped 1 previous similar message [ 4216.302568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4216.313602] Lustre: Skipped 1 previous similar message [ 4221.411149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4222.356677] Lustre: server umount lustre-MDT0000 complete [ 4226.788859] LustreError: 130497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748436 with bad export cookie 17127306320367394661 [ 4226.805149] LustreError: 130497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4226.812234] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4227.208461] Lustre: server umount lustre-MDT0001 complete [ 4236.721452] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 22:34:05 (1788748445) [ 4243.656858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.130040] LustreError: 134887:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4244.145022] LustreError: 134887:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 4244.192285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4248.655710] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4249.579580] LustreError: 134888:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4254.691441] LustreError: 134887:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4259.445538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4259.936330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4264.922272] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4273.066684] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4279.240356] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4283.940158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.957916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4283.973704] Lustre: Skipped 2 previous similar messages [ 4288.456827] Lustre: server umount lustre-MDT0000 complete [ 4288.993757] LustreError: 134887:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4289.012746] LustreError: 134887:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4292.093541] LustreError: 134869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748502 with bad export cookie 17127306320367395550 [ 4292.105957] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4292.106185] LustreError: 134869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4292.572349] Lustre: server umount lustre-MDT0001 complete [ 4300.759324] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 22:35:10 (1788748510) [ 4307.021982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4322.591613] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4327.711584] LustreError: 137301:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 4353.439672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4359.888982] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4367.888524] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4370.264937] Lustre: server umount lustre-OST0000 complete [ 4378.774730] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 22:36:28 (1788748588) [ 4387.350621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4387.743165] LustreError: 138747:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4387.794506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.834189] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4397.027184] LustreError: 138748:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.044592] LustreError: 138748:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4402.660546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4407.859458] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4416.874393] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4424.306059] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4432.344992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4432.987879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4432.993280] Lustre: Skipped 1 previous similar message [ 4439.771314] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4447.783477] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4458.467844] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4458.477694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4458.493035] Lustre: Skipped 3 previous similar messages [ 4458.509418] Lustre: Skipped 2 previous similar messages [ 4463.590938] LustreError: 138742:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.616715] LustreError: 138742:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 4463.640269] Lustre: server umount lustre-MDT0000 complete [ 4467.317162] LustreError: 138729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748677 with bad export cookie 17127306320367396215 [ 4467.328231] LustreError: 138729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4467.329278] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4467.666916] Lustre: server umount lustre-MDT0001 complete [ 4482.721191] Lustre: server umount lustre-OST0000 complete [ 4492.153875] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 22:38:21 (1788748701) [ 4499.433168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.117153] LustreError: 142403:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.240642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4505.752505] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4515.925554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4520.782422] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4529.412559] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4536.090342] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4543.750793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4545.105774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4551.751879] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4559.270595] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4564.078826] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4564.396964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4569.811264] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4570.125943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4575.714876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4575.727985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4575.737125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4576.740225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4578.306673] Lustre: server umount lustre-OST0000 complete [ 4582.880748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4586.980616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.995208] Lustre: Skipped 2 previous similar messages [ 4588.341338] Lustre: server umount lustre-MDT0000 complete [ 4592.099496] LustreError: 145705:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4592.122176] LustreError: 145705:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 4592.761418] LustreError: 145718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748802 with bad export cookie 17127306320367397286 [ 4592.763705] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4592.773168] LustreError: 145718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4593.184073] Lustre: server umount lustre-MDT0001 complete [ 4601.658145] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 22:40:11 (1788748811) [ 4608.014378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4623.585501] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4628.703490] LustreError: 146564:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 4654.367456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4654.378831] Lustre: Skipped 2 previous similar messages [ 4660.777397] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4667.308564] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4674.798148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.280441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4679.049819] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4685.997726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4690.078567] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4696.843099] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4701.582322] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4705.224646] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4705.424719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4709.681553] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4709.904369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4711.925711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4711.932866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4711.941247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4711.943505] Lustre: Skipped 5 previous similar messages [ 4726.239307] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4726.452633] Lustre: server umount lustre-MDT0000 complete [ 4730.339891] LustreError: 146572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748940 with bad export cookie 17127306320367398308 [ 4730.361811] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4730.694553] Lustre: server umount lustre-MDT0001 complete [ 4744.822707] Lustre: server umount lustre-OST0000 complete [ 4752.351965] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 22:42:42 (1788748962) [ 4758.930261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4774.496430] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4779.615453] LustreError: 150757:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 4811.027762] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4817.488608] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4822.851828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4823.157827] LustreError: 150782:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4823.178185] LustreError: 150782:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 4823.287282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4826.628791] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4833.455028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4837.777359] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4846.026165] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4850.589589] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4856.008484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4856.110717] Lustre: lustre-OST0001: new disk, initializing [ 4856.112964] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4856.117856] Lustre: 153629:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4861.977631] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4864.564891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4864.588153] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4864.718230] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4869.590753] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4872.732775] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4872.906950] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4876.412832] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4876.561793] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4879.843114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4879.843408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4879.848188] LustreError: Skipped 1 previous similar message [ 4879.856728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4879.864214] Lustre: Skipped 3 previous similar messages [ 4879.874882] Lustre: Skipped 10 previous similar messages [ 4884.048178] Lustre: server umount lustre-OST0000 complete [ 4901.343179] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4901.509394] Lustre: server umount lustre-OST0001 complete [ 4905.442078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4905.452737] LustreError: Skipped 2 previous similar messages [ 4911.539851] Lustre: server umount lustre-MDT0000 complete [ 4914.698395] LustreError: 150765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749124 with bad export cookie 17127306320367399260 [ 4914.709760] LustreError: 150765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4914.713947] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4915.036368] Lustre: server umount lustre-MDT0001 complete [ 4934.157228] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 22:45:43 (1788749143) [ 4935.552800] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4936.675512] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4937.891763] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 22:45:47 (1788749147) [ 4943.123279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4943.411674] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4943.459943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4943.621551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4943.623957] Lustre: Skipped 6 previous similar messages [ 4947.284509] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4954.221887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4954.285936] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4954.305530] Lustre: 157198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4957.956888] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4964.070605] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4968.647389] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4974.108247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4974.248989] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4979.566284] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4982.367369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4985.138936] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4988.704271] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4988.836275] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4992.617122] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4992.806650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4997.601304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4997.607482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4997.615672] Lustre: Skipped 4 previous similar messages [ 5000.592673] Lustre: server umount lustre-OST0000 complete [ 5019.320746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5023.714315] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.720701] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5024.101654] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5029.330449] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5033.378389] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5038.056875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.066662] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5038.790871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 5042.245772] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5042.541456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5046.941825] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9fda4462b000.ost_server_uuid 50 [ 5048.068537] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9fda4462b000.ost_server_uuid in FULL state after 0 sec [ 5048.185105] LustreError: 157202:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.41@tcp) returned error from blocking AST (req@000000006ad760d1 x1875638526141568 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000003309abcb/0xedb06a5aa0cd3ab6 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.41@tcp remote: 0xf860ae205f4f6651 expref: 5 pid: 157203 timeout: 5148 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5048.218047] LustreError: lustre-MDT0000: A client on nid 192.168.201.41@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5048.225792] LustreError: 157193:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.41@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000003309abcb/0xedb06a5aa0cd3ab6 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.41@tcp remote: 0xf860ae205f4f6651 expref: 6 pid: 157203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5057.456422] Lustre: server umount lustre-OST0000 complete [ 5062.112878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5062.122604] LustreError: Skipped 1 previous similar message [ 5066.599090] Lustre: server umount lustre-MDT0000 complete [ 5069.464952] LustreError: 157190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749279 with bad export cookie 17127306320367400443 [ 5069.467032] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5069.470797] LustreError: 157190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5075.704451] Lustre: server umount lustre-MDT0001 complete [ 5082.153896] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5083.794614] Key type lgssc unregistered [ 5083.985526] LNet: 163339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5083.996847] LNetError: 163339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5084.009298] LNet: Removed LNI 192.168.201.141@tcp [ 5084.605511] Key type .llcrypt unregistered [ 5084.607308] Key type ._llcrypt unregistered [ 5101.186406] Key type ._llcrypt registered [ 5101.188893] Key type .llcrypt registered [ 5101.273403] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5101.934434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5101.959818] alg: No test for adler32 (adler32-zlib) [ 5102.881302] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 5103.013222] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5104.639229] Key type lgssc registered [ 5105.467306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5111.487155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5111.500784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5112.864310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5116.180311] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5121.547902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5121.772106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5124.893976] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5131.078713] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5135.706428] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5141.017432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5141.239312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5145.521669] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5151.143480] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5157.626490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5164.672278] Lustre: Failing over lustre-MDT0000 [ 5164.899726] Lustre: server umount lustre-MDT0000 complete [ 5168.097346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5168.098499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5168.102035] LustreError: 164727:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5168.105385] Lustre: Skipped 2 previous similar messages [ 5168.112931] LustreError: 164727:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 5173.219277] LustreError: 166978:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5173.235474] LustreError: 166978:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5176.375425] LustreError: 164707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.41@tcp arrived at 1788749386 with bad export cookie 6602485366947030028 [ 5178.337286] LustreError: 165454:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5178.353989] LustreError: 165454:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5181.032335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5181.120960] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5181.294055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5181.319290] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5183.845430] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5185.414907] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5185.417211] Lustre: lustre-MDT0000: Denying connection for new client 1142893f-4eec-463a-bedc-3b7d491abb3a (at 192.168.201.41@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 5186.540763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5193.698741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5193.707097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5193.722804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5196.769874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5196.771516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5198.847104] Lustre: server umount lustre-OST0000 complete [ 5201.107655] LustreError: 168475:0:(ldlm_lib.c:3042:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5201.111187] Lustre: 167824:0:(ldlm_lib.c:2442:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5201.118402] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5201.132734] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5201.145106] LustreError: 167824:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f81ab20e x1875639599669632/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5201.160390] LustreError: 167824:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5201.166965] LustreError: 167824:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5201.173579] Lustre: lustre-MDT0000: Recovery over after 0:16, of 2 clients 0 recovered and 2 were evicted. [ 5201.351303] Lustre: server umount lustre-MDT0000 complete [ 5201.890589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5201.896618] LustreError: 166978:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5203.653621] LustreError: 164706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749413 with bad export cookie 6602485366947030357 [ 5203.659701] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5203.665186] LustreError: 164706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5209.875256] Lustre: server umount lustre-MDT0001 complete [ 5216.693736] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5218.499490] Key type lgssc unregistered [ 5218.720976] LNet: 169227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5218.729837] LNetError: 169227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5218.744595] LNet: Removed LNI 192.168.201.141@tcp [ 5219.300438] Key type .llcrypt unregistered [ 5219.303396] Key type ._llcrypt unregistered [ 5233.597709] Key type ._llcrypt registered [ 5233.599261] Key type .llcrypt registered [ 5233.655825] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5234.330619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5234.341614] alg: No test for adler32 (adler32-zlib) [ 5235.320883] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 5235.500229] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5237.128160] Key type lgssc registered [ 5237.797411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5242.207231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5242.216619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5243.465718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5246.173923] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5251.956835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.194567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5254.833266] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5259.583521] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5263.113856] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5267.207392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5267.379712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5268.403312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5270.787961] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5274.792527] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5282.796861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5282.802609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5282.811669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5282.814456] Lustre: Skipped 1 previous similar message [ 5282.823776] Lustre: Skipped 1 previous similar message [ 5284.972918] Lustre: server umount lustre-OST0000 complete [ 5287.907687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.910288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5287.913349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5287.920705] Lustre: Skipped 1 previous similar message [ 5293.024215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5293.028956] Lustre: Skipped 1 previous similar message [ 5298.144454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5298.146941] Lustre: Skipped 1 previous similar message [ 5301.216474] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5301.305263] Lustre: server umount lustre-MDT0000 complete [ 5302.242087] LustreError: 170609:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5302.250273] LustreError: 170609:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 5303.415084] LustreError: 170595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749513 with bad export cookie 7884740132081427014 [ 5303.417602] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5303.418963] LustreError: 170595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5303.623532] Lustre: server umount lustre-MDT0001 complete [ 5310.263744] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5311.958947] Key type lgssc unregistered [ 5312.197671] LNet: 174143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5312.204989] LNetError: 174143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5312.223308] LNet: Removed LNI 192.168.201.141@tcp [ 5312.767467] Key type .llcrypt unregistered [ 5312.769418] Key type ._llcrypt unregistered [ 5342.906813] Key type ._llcrypt registered [ 5342.911330] Key type .llcrypt registered [ 5342.995590] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5344.847803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5344.882960] alg: No test for adler32 (adler32-zlib) [ 5346.176051] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 5346.582894] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5348.367274] Key type lgssc registered [ 5350.947716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5367.443605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5367.511562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5370.162945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5379.344563] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5380.452682] LustreError: 175843:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5380.482766] LustreError: 175843:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5385.700832] LustreError: 175844:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5390.823530] LustreError: 175843:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5395.433515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5396.117218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5401.829479] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5409.152290] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5414.578860] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5421.257528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5421.505440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5431.113218] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5442.047323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5443.319826] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5452.258425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.263166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5452.266207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5452.292376] Lustre: Skipped 1 previous similar message [ 5452.332132] Lustre: Skipped 1 previous similar message [ 5457.382536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5458.038493] Lustre: server umount lustre-OST0000 complete [ 5462.496978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5462.500708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5462.505662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5462.516133] Lustre: Skipped 1 previous similar message [ 5462.520584] Lustre: Skipped 1 previous similar message [ 5467.617849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5467.630125] Lustre: Skipped 2 previous similar messages [ 5472.739384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5472.745696] Lustre: Skipped 1 previous similar message [ 5476.319286] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5476.545176] Lustre: server umount lustre-MDT0000 complete [ 5477.858456] LustreError: 176575:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5477.888270] LustreError: 176575:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5480.571711] LustreError: 175823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749690 with bad export cookie 11246181470690559614 [ 5480.580295] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5480.583253] LustreError: 175823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5481.053523] Lustre: server umount lustre-MDT0001 complete [ 5492.234949] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5494.988879] Key type lgssc unregistered [ 5495.239774] LNet: 179380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5495.255570] LNetError: 179380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5495.279364] LNet: Removed LNI 192.168.201.141@tcp [ 5496.249203] Key type .llcrypt unregistered [ 5496.251662] Key type ._llcrypt unregistered [ 5518.335309] Key type ._llcrypt registered [ 5518.345249] Key type .llcrypt registered [ 5518.526455] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5519.833412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5519.948013] alg: No test for adler32 (adler32-zlib) [ 5520.921990] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 5521.166872] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5522.824851] Key type lgssc registered [ 5524.064969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5535.306353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5535.325599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5536.733803] Lustre: *** cfs_fail_loc=135, val=0*** [ 5536.822566] LustreError: 180903:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5536.839510] LustreError: 180903:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.141@tcp: cfg command failed: rc = -2 [ 5536.852445] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5536.852445] [ 5536.861443] LustreError: MGC192.168.201.141@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5536.883984] LustreError: 180864:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5536.893862] LustreError: 180864:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5536.901314] LustreError: 180864:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5536.907619] LustreError: 180864:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5537.001720] Lustre: server umount lustre-MDT0000 complete [ 5537.009104] LustreError: 180864:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5544.172261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5544.538874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5549.489492] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5554.665701] LustreError: 181274:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5554.680154] LustreError: 181274:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5558.796579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5559.142569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5563.888227] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5573.409298] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5579.395501] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5584.481908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5584.498587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5584.504664] Lustre: Skipped 1 previous similar message [ 5584.526658] Lustre: Skipped 1 previous similar message [ 5587.824160] Lustre: server umount lustre-MDT0000 complete [ 5589.985239] LustreError: 181271:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5590.010835] LustreError: 181271:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5591.720094] LustreError: 181255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749801 with bad export cookie 2108330015886871041 [ 5591.728967] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5591.753231] LustreError: 181255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5592.082371] Lustre: server umount lustre-MDT0001 complete [ 5603.012947] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5605.998142] Key type lgssc unregistered [ 5606.330576] LNet: 183594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5606.347320] LNetError: 183594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5606.374959] LNet: Removed LNI 192.168.201.141@tcp [ 5607.417405] Key type .llcrypt unregistered [ 5607.419805] Key type ._llcrypt unregistered [ 5634.116391] Key type ._llcrypt registered [ 5634.125033] Key type .llcrypt registered [ 5634.312804] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5634.999222] Key type .llcrypt unregistered [ 5635.003823] Key type ._llcrypt unregistered [ 5656.892865] Key type ._llcrypt registered [ 5656.899975] Key type .llcrypt registered [ 5657.152980] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5658.877413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5659.250231] alg: No test for adler32 (adler32-zlib) [ 5660.627713] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 5661.169658] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5662.967355] Key type lgssc registered [ 5664.412993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5674.549673] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5674.563257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5690.080359] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5695.199384] LustreError: 185722:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 5721.887682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5727.720039] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5735.179835] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5741.967757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5742.364644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5742.427794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5746.135650] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5747.685169] LustreError: 185747:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5747.702649] LustreError: 185747:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5752.804374] LustreError: 185748:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5754.055427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5754.381603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5759.253278] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5766.461099] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5771.501987] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5774.124704] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5785.060928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5785.068965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5785.080649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5787.114166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5787.118438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5792.228055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5792.232871] Lustre: Skipped 2 previous similar messages [ 5797.349603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5797.358139] Lustre: Skipped 1 previous similar message [ 5799.391461] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5799.603173] Lustre: server umount lustre-OST0000 complete [ 5805.537887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5805.542680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5805.552148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5807.586705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5809.708153] Lustre: server umount lustre-MDT0000 complete [ 5812.705995] LustreError: 186764:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5812.728518] LustreError: 186764:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5813.211593] LustreError: 185728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750023 with bad export cookie 2190867695494421014 [ 5813.219988] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5813.223864] LustreError: 185728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5813.689640] Lustre: server umount lustre-MDT0001 complete [ 5825.638996] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5828.088863] Key type lgssc unregistered [ 5828.360237] LNet: 189685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5828.364781] LNetError: 189685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5828.375893] LNet: Removed LNI 192.168.201.141@tcp [ 5829.594715] Key type .llcrypt unregistered [ 5829.596193] Key type ._llcrypt unregistered [ 5854.393632] Key type ._llcrypt registered [ 5854.399672] Key type .llcrypt registered [ 5854.491112] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5855.571805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5855.627814] alg: No test for adler32 (adler32-zlib) [ 5856.757633] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 5857.089951] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5858.791523] Key type lgssc registered [ 5859.804610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5866.897492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5866.912698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5868.336538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5872.310611] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5878.753472] LustreError: 191109:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5878.778364] LustreError: 191109:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5879.777513] LustreError: 191108:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5880.819349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5881.178008] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5885.688637] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5893.150314] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5899.207445] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5906.556664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5906.890718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5912.727094] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5914.025562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5920.191504] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5925.817398] Lustre: Failing over lustre-MDT0000 [ 5926.115544] Lustre: server umount lustre-MDT0000 complete [ 5927.402824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5927.408535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5927.420786] LustreError: 191109:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.440270] LustreError: 191108:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.445503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5929.464416] LustreError: 191108:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5929.476563] Lustre: Skipped 1 previous similar message [ 5934.564158] LustreError: 191835:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.579644] LustreError: 191835:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5943.473089] LustreError: 191105:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5943.494899] LustreError: 191105:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 5944.756851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.945652] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5945.235084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5945.306404] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5948.588198] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5949.849077] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5950.454942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5950.470044] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5950.533851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5954.447172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5960.383676] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5966.304912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5966.315089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.336262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5968.086370] Lustre: server umount lustre-OST0000 complete [ 5976.038527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5976.042061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5976.055324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5976.064185] Lustre: Skipped 1 previous similar message [ 5978.023096] Lustre: server umount lustre-MDT0000 complete [ 5981.159462] LustreError: 191103:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5981.186506] LustreError: 191103:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 5982.460672] LustreError: 191089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750192 with bad export cookie 13607010888634986959 [ 5982.464193] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5982.470950] LustreError: 191089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5993.208924] Lustre: server umount lustre-MDT0001 complete [ 6009.634262] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6012.230516] Key type lgssc unregistered [ 6012.480554] LNet: 195946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6012.484846] LNetError: 195946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6012.503128] LNet: Removed LNI 192.168.201.141@tcp [ 6013.393320] Key type .llcrypt unregistered [ 6013.395930] Key type ._llcrypt unregistered [ 6039.700615] Key type ._llcrypt registered [ 6039.704085] Key type .llcrypt registered [ 6039.850218] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6040.905435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6040.964137] alg: No test for adler32 (adler32-zlib) [ 6042.029914] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 6042.270089] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6043.943806] Key type lgssc registered [ 6044.987521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6051.909825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6051.934619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6053.643691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6058.546799] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6063.078533] LustreError: 197356:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6063.103596] LustreError: 197356:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6068.198255] LustreError: 197355:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6068.700695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.247891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6074.452896] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6083.074836] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6089.623544] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6097.005657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6097.362620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6098.428625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6103.341966] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6111.274560] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6115.313588] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6122.966439] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6143.249750] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6159.843444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6159.854155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6159.876164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6160.867225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6160.868423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6160.879445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.904813] Lustre: Skipped 1 previous similar message [ 6164.309194] Lustre: server umount lustre-OST0000 complete [ 6171.120992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6171.128451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6171.140935] Lustre: Skipped 1 previous similar message [ 6171.154981] Lustre: Skipped 1 previous similar message [ 6176.226747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6176.239725] Lustre: Skipped 1 previous similar message [ 6181.348493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6181.358635] Lustre: Skipped 1 previous similar message [ 6182.367224] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6182.768807] Lustre: server umount lustre-MDT0000 complete [ 6186.466362] LustreError: 200185:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6186.494484] LustreError: 200185:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6187.519661] LustreError: 197338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750397 with bad export cookie 7159465263378114549 [ 6187.521494] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6187.533409] LustreError: 197338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6187.827859] Lustre: server umount lustre-MDT0001 complete [ 6199.312641] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6202.756810] Key type lgssc unregistered [ 6203.042761] LNet: 201253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6203.057657] LNetError: 201253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6203.077020] LNet: Removed LNI 192.168.201.141@tcp [ 6204.184941] Key type .llcrypt unregistered [ 6204.187274] Key type ._llcrypt unregistered [ 6225.016557] Key type ._llcrypt registered [ 6225.018231] Key type .llcrypt registered [ 6225.148722] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6225.942209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6225.960698] alg: No test for adler32 (adler32-zlib) [ 6227.183455] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 6227.429259] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6229.071310] Key type lgssc registered [ 6230.111810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6238.808367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6238.834059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6240.412878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6245.317515] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6249.957651] LustreError: 202517:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6250.016219] LustreError: 202517:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6254.001214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.299813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6258.788049] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6266.756619] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6272.847838] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6279.625351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6279.954782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6285.120319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6285.916519] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6294.563765] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6300.581905] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6314.210957] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6323.586568] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6329.131570] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6340.579902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6340.601604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6340.603292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6340.617203] Lustre: Skipped 1 previous similar message [ 6342.969318] Lustre: server umount lustre-OST0000 complete [ 6346.721748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6346.731167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6346.743747] Lustre: Skipped 1 previous similar message [ 6346.749256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6350.817482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6350.821650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6352.874556] Lustre: server umount lustre-MDT0000 complete [ 6355.942858] LustreError: 203554:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.959475] LustreError: 203554:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6357.314811] LustreError: 205640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750567 with bad export cookie 11660325408089057489 [ 6357.316573] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6357.322541] LustreError: 205640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6357.744213] Lustre: server umount lustre-MDT0001 complete [ 6369.247696] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6372.240866] Key type lgssc unregistered [ 6372.558680] LNet: 207395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6372.574231] LNetError: 207395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6372.597472] LNet: Removed LNI 192.168.201.141@tcp [ 6373.703688] Key type .llcrypt unregistered [ 6373.710996] Key type ._llcrypt unregistered [ 6399.590776] Key type ._llcrypt registered [ 6399.595042] Key type .llcrypt registered [ 6399.781755] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6400.854396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6400.987331] alg: No test for adler32 (adler32-zlib) [ 6402.200850] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 6402.418839] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6404.175250] Key type lgssc registered [ 6405.646146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6413.976354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6413.989404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6415.610500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6420.391955] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6425.061768] LustreError: 208805:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6425.082841] LustreError: 208805:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6429.556571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6430.005460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6434.249301] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6440.963371] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6446.736674] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6453.376746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6453.873453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6460.015262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6460.125839] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6468.706959] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6485.993813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6486.001994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6486.004058] LustreError: Skipped 1 previous similar message [ 6486.009959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6486.033750] Lustre: Skipped 1 previous similar message [ 6491.104448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6491.111427] Lustre: Skipped 1 previous similar message [ 6491.444346] Lustre: server umount lustre-OST0000 complete [ 6496.237681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.251652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6496.266079] Lustre: Skipped 1 previous similar message [ 6496.281984] Lustre: Skipped 1 previous similar message [ 6501.139669] Lustre: server umount lustre-MDT0000 complete [ 6501.351197] LustreError: 210046:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6501.368442] LustreError: 210046:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6504.721047] LustreError: 208785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750714 with bad export cookie 945363409201010654 [ 6504.736601] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6504.740375] LustreError: 208785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6505.127632] Lustre: server umount lustre-MDT0001 complete [ 6516.148981] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6518.753612] Key type lgssc unregistered [ 6519.053073] LNet: 212490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6519.059230] LNetError: 212490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6520.109696] LNet: Removed LNI 192.168.201.141@tcp [ 6520.990660] Key type .llcrypt unregistered [ 6520.995471] Key type ._llcrypt unregistered [ 6540.595203] Key type ._llcrypt registered [ 6540.597821] Key type .llcrypt registered [ 6540.707547] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6542.362863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6542.404667] alg: No test for adler32 (adler32-zlib) [ 6543.441486] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 6543.769280] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6545.535158] Key type lgssc registered [ 6546.803580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6555.519625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6555.559478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6557.195866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6561.843886] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6567.397246] LustreError: 213758:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.432322] LustreError: 213758:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6570.850798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.385550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6577.056391] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6585.945562] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6592.517111] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6598.711829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6599.172057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6602.296264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6606.045926] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6613.260873] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6631.910935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6631.913212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6631.949746] Lustre: Skipped 1 previous similar message [ 6634.455860] Lustre: server umount lustre-OST0000 complete [ 6642.144784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6642.156889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6642.171873] Lustre: Skipped 1 previous similar message [ 6642.184729] Lustre: Skipped 2 previous similar messages [ 6644.928592] Lustre: server umount lustre-MDT0000 complete [ 6647.269237] LustreError: 213757:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6647.284989] LustreError: 213757:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6648.860887] LustreError: 215604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750858 with bad export cookie 6741424517575170964 [ 6648.864210] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6648.883376] LustreError: 215604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6649.557943] Lustre: server umount lustre-MDT0001 complete [ 6661.575426] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6665.068844] Key type lgssc unregistered [ 6665.392150] LNet: 217440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6665.398315] LNetError: 217440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6665.413245] LNet: Removed LNI 192.168.201.141@tcp [ 6666.473799] Key type .llcrypt unregistered [ 6666.475934] Key type ._llcrypt unregistered [ 6688.102928] Key type ._llcrypt registered [ 6688.106823] Key type .llcrypt registered [ 6688.210477] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6689.924538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6689.955749] alg: No test for adler32 (adler32-zlib) [ 6691.130819] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 6691.399862] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6693.095188] Key type lgssc registered [ 6694.326558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6702.485658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6702.503725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6703.866351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6708.445784] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6713.313118] LustreError: 218702:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6713.339146] LustreError: 218702:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6717.251761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.756311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6722.561776] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6730.339611] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6736.684296] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6743.282142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6743.715441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6748.855245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6750.532927] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6758.776104] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6769.070895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6769.127578] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6769.328943] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6777.243924] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6785.061458] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6786.723733] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6798.436935] Lustre: Permanently deactivating lustre-OST0001 [ 6798.450648] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6808.186088] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6817.977666] Lustre: Permanently reactivating lustre-OST0001 [ 6818.025857] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6818.032159] Lustre: Skipped 2 previous similar messages [ 6826.606238] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6826.620583] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6826.627774] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6826.648351] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6826.674323] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6829.498524] Lustre: server umount lustre-OST0001 complete [ 6830.580401] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6830.595685] Lustre: Skipped 1 previous similar message [ 6830.600945] LustreError: 220527:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6830.629554] LustreError: 220527:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6834.144278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6834.159309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6834.176609] Lustre: Skipped 1 previous similar message [ 6834.180224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6835.684465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6835.685980] LustreError: 220527:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6835.727774] LustreError: 220527:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6839.717898] Lustre: server umount lustre-OST0000 complete [ 6845.920884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6845.923687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6845.934855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6845.955724] Lustre: Skipped 2 previous similar messages [ 6845.969485] Lustre: Skipped 2 previous similar messages [ 6850.661483] Lustre: server umount lustre-MDT0000 complete [ 6851.042066] LustreError: 218702:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6851.062434] LustreError: 218702:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6854.594744] LustreError: 222324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751064 with bad export cookie 5755721578075057186 [ 6854.597134] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6854.618562] LustreError: 222324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6855.079945] Lustre: server umount lustre-MDT0001 complete [ 6866.842835] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6870.245649] Key type lgssc unregistered [ 6870.603785] LNet: 223796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6870.615566] LNetError: 223796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6870.631747] LNet: Removed LNI 192.168.201.141@tcp [ 6871.675187] Key type .llcrypt unregistered [ 6871.682635] Key type ._llcrypt unregistered [ 6914.017654] Key type ._llcrypt registered [ 6914.019882] Key type .llcrypt registered [ 6914.171144] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6915.446411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6915.589061] alg: No test for adler32 (adler32-zlib) [ 6916.791147] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 6917.069647] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6918.783441] Key type lgssc registered [ 6920.127477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6928.472565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6928.501949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6929.814552] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6929.840139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6930.035404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6934.666862] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6943.621110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6943.726579] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6943.765537] Lustre: 226000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6944.046649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6948.649646] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6956.320034] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6962.027665] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6968.100995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6968.290069] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6968.556957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6974.359855] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6977.260764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6982.262170] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6986.538528] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6994.860443] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7005.184905] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7014.495749] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7023.186858] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7048.281761] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7048.293116] Lustre: Skipped 2 previous similar messages [ 7087.876673] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7087.882449] Lustre: Skipped 4 previous similar messages [ 7156.430554] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7156.435852] Lustre: Skipped 7 previous similar messages [ 7177.187428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7177.191325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7177.213311] LustreError: Skipped 1 previous similar message [ 7177.234761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7177.256996] Lustre: Skipped 1 previous similar message [ 7181.781816] Lustre: server umount lustre-OST0000 complete [ 7187.425388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7187.440490] Lustre: Skipped 1 previous similar message [ 7187.446071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7187.453289] Lustre: Skipped 1 previous similar message [ 7191.815741] Lustre: server umount lustre-MDT0000 complete [ 7195.297376] LustreError: 229057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751405 with bad export cookie 3302408831883604525 [ 7195.297776] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7195.302406] LustreError: 229057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7195.821937] Lustre: server umount lustre-MDT0001 complete [ 7206.618862] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7209.067874] Key type lgssc unregistered [ 7209.365508] LNet: 230795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7209.372213] LNetError: 230795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7209.406773] LNet: Removed LNI 192.168.201.141@tcp [ 7210.275581] Key type .llcrypt unregistered [ 7210.282948] Key type ._llcrypt unregistered [ 7232.881277] Key type ._llcrypt registered [ 7232.883688] Key type .llcrypt registered [ 7232.966568] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7233.765064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7233.781208] alg: No test for adler32 (adler32-zlib) [ 7234.775497] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 7234.947915] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 7236.592001] Key type lgssc registered [ 7237.532580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7244.019862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7244.053591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7245.410269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7249.585316] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7255.009683] LustreError: 232204:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7255.032682] LustreError: 232204:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7257.993770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7258.472784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7262.399994] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7270.513043] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7277.142613] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7284.366826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7284.821708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7289.970578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7291.445705] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7299.264871] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7319.013982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7319.019847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7319.020911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7319.024620] Lustre: Skipped 1 previous similar message [ 7321.022157] Lustre: server umount lustre-OST0000 complete [ 7325.157839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7325.167476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7325.195116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7325.197809] Lustre: Skipped 1 previous similar message [ 7329.255949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7329.260395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7329.287733] Lustre: Skipped 1 previous similar message [ 7331.238870] Lustre: server umount lustre-MDT0000 complete [ 7334.380130] LustreError: 232931:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7335.502545] LustreError: 232183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751545 with bad export cookie 7802617075800487845 [ 7335.506842] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7335.514359] LustreError: 232183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7335.941449] Lustre: server umount lustre-MDT0001 complete [ 7347.551429] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7350.225468] Key type lgssc unregistered [ 7350.651696] LNet: 235893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7350.677704] LNetError: 235893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7350.701647] LNet: Removed LNI 192.168.201.141@tcp [ 7351.638417] Key type .llcrypt unregistered [ 7351.641677] Key type ._llcrypt unregistered [ 7372.881472] Key type ._llcrypt registered [ 7372.883414] Key type .llcrypt registered [ 7372.968134] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7373.889789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7373.921204] alg: No test for adler32 (adler32-zlib) [ 7375.051577] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 7375.330293] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 7376.983228] Key type lgssc registered [ 7377.913721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7648.793693] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7651.708767] Key type lgssc unregistered [ 7652.043671] LNet: 237911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7652.048184] LNetError: 237911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7652.079636] LNet: Removed LNI 192.168.201.141@tcp [ 7653.123480] Key type .llcrypt unregistered [ 7653.125795] Key type ._llcrypt unregistered [ 7679.303688] Key type ._llcrypt registered [ 7679.305723] Key type .llcrypt registered [ 7679.417183] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 7695.185471] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7696.111978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7696.217213] alg: No test for adler32 (adler32-zlib) [ 7697.325439] Lustre: Lustre: Build Version: 2.17.58_40_g869f601 [ 7697.612852] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 7699.327373] Key type lgssc registered [ 7700.425632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7731.437959] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7435 sec ========== 23:32:20 (1788751940) [ 7732.891584] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:32:22 (1788751942) === [ 7736.066508] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:32:25 (1788751945) === [ 7776.577557] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7779.060788] Key type lgssc unregistered [ 7779.366567] LNet: 243142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7779.371852] LNetError: 243142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7779.381918] LNet: Removed LNI 192.168.201.141@tcp [ 7780.175054] Key type .llcrypt unregistered [ 7780.176846] Key type ._llcrypt unregistered