[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 2743316732 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002012] Switched APIC routing to physical x2apic. [ 0.003000] kvm-guest: setup PV IPIs [ 0.003000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.003000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.003000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.003000] pid_max: default: 32768 minimum: 301 [ 0.003000] LSM: Security Framework initializing [ 0.003000] Yama: becoming mindful. [ 0.003000] SELinux: Initializing. [ 0.003000] *** VALIDATE selinux *** [ 0.011284] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.016814] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.017293] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.018115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.019128] *** VALIDATE tmpfs *** [ 0.021513] *** VALIDATE proc *** [ 0.023073] *** VALIDATE cgroup *** [ 0.024009] *** VALIDATE cgroup2 *** [ 0.026103] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.027151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.028008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.029027] Spectre V2 : User space: Vulnerable [ 0.030009] Speculative Store Bypass: Vulnerable [ 0.033570] debug: unmapping init [mem 0xffffffff9ee59000-0xffffffff9ee60fff] [ 0.035268] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.036743] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.037023] ... version: 2 [ 0.038010] ... bit width: 48 [ 0.039010] ... generic registers: 4 [ 0.040010] ... value mask: 0000ffffffffffff [ 0.041018] ... max period: 00007fffffffffff [ 0.042011] ... fixed-purpose events: 3 [ 0.043008] ... event mask: 000000070000000f [ 0.044318] rcu: Hierarchical SRCU implementation. [ 0.046548] smp: Bringing up secondary CPUs ... [ 0.047577] x86: Booting SMP configuration: [ 0.048035] .... node #0, CPUs: #1 #2 #3 [ 0.057348] smp: Brought up 1 node, 4 CPUs [ 0.059011] smpboot: Max logical packages: 1 [ 0.060012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.097564] node 0 deferred pages initialised in 35ms [ 0.102008] devtmpfs: initialized [ 0.103242] x86/mm: Memory block size: 128MB [ 0.106361] gcov: version magic: 0x41383552 [ 0.108727] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.109095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.110307] pinctrl core: initialized pinctrl subsystem [ 0.111371] [ 0.112008] ************************************************************* [ 0.113016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.114029] ** ** [ 0.115016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.116015] ** ** [ 0.117015] ** This means that this kernel is built to expose internal ** [ 0.118020] ** IOMMU data structures, which may compromise security on ** [ 0.119016] ** your system. ** [ 0.120028] ** ** [ 0.121024] ** If you see this message and you are not debugging the ** [ 0.122015] ** kernel, report this immediately to your vendor! ** [ 0.123019] ** ** [ 0.124020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.125014] ************************************************************* [ 0.127147] NET: Registered protocol family 16 [ 0.128498] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.129161] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.130239] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.132054] cpuidle: using governor menu [ 0.134184] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.137669] PCI: Using configuration type 1 for base access [ 0.140158] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.151367] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.153085] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.156181] cryptd: max_cpu_qlen set to 1000 [ 0.158210] ACPI: Added _OSI(Module Device) [ 0.160036] ACPI: Added _OSI(Processor Device) [ 0.162013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.165017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.173685] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.184299] ACPI: Interpreter enabled [ 0.185078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.186076] ACPI: Using IOAPIC for interrupt routing [ 0.187130] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.189084] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.209471] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.210042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.211039] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.212094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.217685] acpiphp: Slot [2] registered [ 0.219087] acpiphp: Slot [5] registered [ 0.220956] acpiphp: Slot [6] registered [ 0.221204] acpiphp: Slot [7] registered [ 0.222162] acpiphp: Slot [8] registered [ 0.223198] acpiphp: Slot [9] registered [ 0.224747] acpiphp: Slot [10] registered [ 0.225674] acpiphp: Slot [3] registered [ 0.226204] acpiphp: Slot [4] registered [ 0.227162] acpiphp: Slot [11] registered [ 0.228137] acpiphp: Slot [12] registered [ 0.230087] acpiphp: Slot [13] registered [ 0.231169] acpiphp: Slot [14] registered [ 0.232154] acpiphp: Slot [15] registered [ 0.234152] acpiphp: Slot [16] registered [ 0.235305] acpiphp: Slot [17] registered [ 0.236820] acpiphp: Slot [18] registered [ 0.237130] acpiphp: Slot [19] registered [ 0.238154] acpiphp: Slot [20] registered [ 0.239881] acpiphp: Slot [21] registered [ 0.240864] acpiphp: Slot [22] registered [ 0.241590] acpiphp: Slot [23] registered [ 0.243175] acpiphp: Slot [24] registered [ 0.244929] acpiphp: Slot [25] registered [ 0.245145] acpiphp: Slot [26] registered [ 0.246147] acpiphp: Slot [27] registered [ 0.247817] acpiphp: Slot [28] registered [ 0.248161] acpiphp: Slot [29] registered [ 0.249454] acpiphp: Slot [30] registered [ 0.250498] acpiphp: Slot [31] registered [ 0.251077] PCI host bridge to bus 0000:00 [ 0.252029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.253031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.254033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.255291] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.256041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.257376] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.258637] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.262076] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.267225] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.280017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.285087] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.286022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.287020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.289029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.293524] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.296792] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.297155] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.300224] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.303018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.312022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.315022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.328010] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.331025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.334017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.343017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.349235] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.352015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.355017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.362029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.368040] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.371013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.374015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.381016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.388509] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.391018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.394018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.401023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.407434] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.410021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.413019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.420018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.426074] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.429020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.432025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.439020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.450434] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.451386] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.452324] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.453335] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.454321] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.495046] iommu: Default domain type: Passthrough [ 0.497044] SCSI subsystem initialized [ 0.498147] ACPI: bus type USB registered [ 0.499188] usbcore: registered new interface driver usbfs [ 0.500090] usbcore: registered new interface driver hub [ 0.501079] usbcore: registered new device driver usb [ 0.502157] pps_core: LinuxPPS API ver. 1 registered [ 0.503009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.504062] PTP clock support registered [ 0.529148] EDAC MC: Ver: 3.0.0 [ 0.531233] PCI: Using ACPI for IRQ routing [ 0.533235] NetLabel: Initializing [ 0.534009] NetLabel: domain hash size = 128 [ 0.535008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.536096] NetLabel: unlabeled traffic allowed by default [ 0.538073] vgaarb: loaded [ 0.539394] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.540012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.546976] clocksource: Switched to clocksource kvm-clock [ 1.040694] VFS: Disk quotas dquot_6.6.0 [ 1.042163] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.053376] *** VALIDATE ramfs *** [ 1.061417] *** VALIDATE hugetlbfs *** [ 1.068238] pnp: PnP ACPI init [ 1.086809] pnp: PnP ACPI: found 6 devices [ 1.114112] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.117281] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.119345] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.121315] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.123551] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.125534] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.128326] NET: Registered protocol family 2 [ 1.131142] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.136565] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.139774] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.145262] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.148757] TCP: Hash tables configured (established 65536 bind 65536) [ 1.151325] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.154136] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.156840] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.159493] NET: Registered protocol family 1 [ 1.161814] RPC: Registered named UNIX socket transport module. [ 1.163720] RPC: Registered udp transport module. [ 1.165397] RPC: Registered tcp transport module. [ 1.167122] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.169609] NET: Registered protocol family 44 [ 1.171712] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.173697] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.175557] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.177987] PCI: CLS 0 bytes, default 64 [ 1.179577] Unpacking initramfs... [ 3.050778] debug: unmapping init [mem 0xffff8fdd7cc54000-0xffff8fdd7ffbffff] [ 3.056272] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.059658] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.064522] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.869379] Initialise system trusted keyrings [ 3.871068] Key type blacklist registered [ 3.873107] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.886205] zbud: loaded [ 3.890133] *** VALIDATE nfs *** [ 3.901292] *** VALIDATE nfs4 *** [ 3.902824] pstore: using deflate compression [ 3.907268] Platform Keyring initialized [ 4.157250] NET: Registered protocol family 38 [ 4.159586] Key type asymmetric registered [ 4.161589] Asymmetric key parser 'x509' registered [ 4.163625] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.167511] io scheduler mq-deadline registered [ 4.169091] io scheduler kyber registered [ 4.170816] io scheduler bfq registered [ 4.172756] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.176296] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.181364] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.195620] ACPI: Power Button [PWRF] [ 4.210164] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.356581] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.600583] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.607825] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.628969] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.659609] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.689459] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.694842] Non-volatile memory driver v1.3 [ 4.696640] Linux agpgart interface v0.103 [ 4.746713] virtio_blk virtio1: [vda] 145920 512-byte logical blocks (74.7 MB/71.3 MiB) [ 4.751870] vda: detected capacity change from 0 to 74711040 [ 4.770847] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.774837] vdb: detected capacity change from 0 to 1073741824 [ 4.806793] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.812132] vdc: detected capacity change from 0 to 2621440000 [ 4.844668] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.850513] vdd: detected capacity change from 0 to 2621440000 [ 4.875579] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.882182] vde: detected capacity change from 0 to 4294967296 [ 4.911190] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.915300] vdf: detected capacity change from 0 to 4294967296 [ 4.928262] libphy: Fixed MDIO Bus: probed [ 4.936951] usbcore: registered new interface driver usbserial_generic [ 4.939846] usbserial: USB Serial support registered for generic [ 4.943318] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.953044] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.956317] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.962538] mousedev: PS/2 mouse device common for all mice [ 4.969659] rtc_cmos 00:05: RTC can wake from S4 [ 4.977177] rtc_cmos 00:05: registered as rtc0 [ 4.979883] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.985381] intel_pstate: CPU model not supported [ 4.993584] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.006643] hid: raw HID events driver (C) Jiri Kosina [ 5.011700] usbcore: registered new interface driver usbhid [ 5.017282] usbhid: USB HID core driver [ 5.021065] drop_monitor: Initializing network drop monitor service [ 5.026365] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.033023] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.046273] Initializing XFRM netlink socket [ 5.067356] NET: Registered protocol family 10 [ 5.071536] Segment Routing with IPv6 [ 5.073726] NET: Registered protocol family 17 [ 5.077935] mpls_gso: MPLS GSO support [ 5.086889] RAS: Correctable Errors collector initialized. [ 5.097933] AVX version of gcm_enc/dec engaged. [ 5.112884] AES CTR mode by8 optimization enabled [ 5.277253] sched_clock: Marking stable (5277203632, 0)->(8267958487, -2990754855) [ 5.287185] registered taskstats version 1 [ 5.293513] Loading compiled-in X.509 certificates [ 5.301654] zswap: loaded using pool lzo/zbud [ 5.495924] Key type big_key registered [ 5.568510] Key type encrypted registered [ 5.578932] ima: No TPM chip found, activating TPM-bypass! [ 5.594942] ima: Allocated hash algorithm: sha1 [ 5.600038] ima: No architecture policies found [ 5.612584] evm: Initialising EVM extended attributes: [ 5.616081] evm: security.selinux [ 5.620077] evm: security.ima [ 5.626685] evm: security.capability [ 5.630549] evm: HMAC attrs: 0x1 [ 5.642646] rtc_cmos 00:05: setting system clock to 2026-08-17 06:10:13 UTC (1786947013) [ 5.679634] debug: unmapping init [mem 0xffffffff9fe03000-0xffffffff9fffffff] [ 5.684884] debug: unmapping init [mem 0xffffffff9eb82000-0xffffffff9ee58fff] [ 5.691330] Write protecting the kernel read-only data: 28672k [ 5.698907] debug: unmapping init [mem 0xffffffff9d203000-0xffffffff9d3fffff] [ 5.703383] debug: unmapping init [mem 0xffffffff9db14000-0xffffffff9dbfffff] [ 5.757796] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.773755] systemd[1]: Detected virtualization kvm. [ 5.777232] systemd[1]: Detected architecture x86-64. [ 5.778913] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.816946] systemd[1]: No hostname configured. [ 5.822514] systemd[1]: Set hostname to . [ 5.828921] random: systemd: uninitialized urandom read (16 bytes read) [ 5.837559] systemd[1]: Initializing machine ID from random generator. [ 9.000177] random: systemd: uninitialized urandom read (16 bytes read) [ 9.005949] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 9.017330] random: systemd: uninitialized urandom read (16 bytes read) [ 9.020602] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 9.103559] random: systemd: uninitialized urandom read (16 bytes read) [ 9.107676] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.853599] device-mapper: uevent: version 1.0.3 [ 10.861915] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [[ 13.023878] random: fast init done  OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 13.137074] virtio_net virtio0 ens2: renamed from eth0 [ 14.194852] scsi host0: ata_piix [ 14.301407] scsi host1: ata_piix [ 14.315493] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 14.322547] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 19.226367] random: crng init done [ 19.232440] random: 6 urandom warning(s) missed due to ratelimiting [ 23.215772] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 26.546169] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 33.472603] printk: systemd: 17 output lines suppressed due to ratelimiting [ 34.218562] SELinux: Disabled at runtime. [ 34.297610] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 34.353277] systemd[1]: Detected virtualization kvm. [ 34.356949] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 38.335491] systemd[1]: initrd-switch-root.service: Succeeded. [ 38.347244] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 38.410520] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 38.434292] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 38.441239] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 38.500971] systemd[1]: Starting Journal Service... Starting Journal Service... [ 38.556472] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ 39.352040] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ 39.427924] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ 39.482799] systemd[1]: Stopped target Initrd Root File System. [ OK ] Stopped target Initrd Root File System. [ 39.539294] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ 39.710285] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 39.743315] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 39.760705] systemd[1]: Stopped target Initrd File Systems. [ OK ] Stopped target Initrd File Systems. [ 39.784891] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 39.811967] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 39.849097] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ 39.870890] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. Starting udev Coldplug all Devices... [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ 40.462657] hrtimer: interrupt took 4819271 ns [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 43.073927] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 46.399485] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 46.423710] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 48.005256] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 48.094776] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (13s / no limit) [** ] A start job is running for Configur…only root support (13s / no limit) [*** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ ***] A start job is running for Configur…only root support (15s / no limit) [ **] A start job is running for Configur…only root support (16s / no limit) [ *] A start job is running for Configur…only root support (16s / no limit)[ 54.854485] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (17s / no limit)[ 55.484676] NFS: Registering the id_resolver key type [ 55.494350] Key type id_resolver registered [ 55.498339] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (17s / no limit) [ *** ] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (18s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg222-server login: [ 122.269467] libcfs: loading out-of-tree module taints kernel. [ 122.319228] Key type ._llcrypt registered [ 122.322498] Key type .llcrypt registered [ 122.436346] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 142.415970] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 144.109422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 144.120625] alg: No test for adler32 (adler32-zlib) [ 145.714781] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 146.858052] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 148.703783] Key type lgssc registered [ 150.456751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 168.580959] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 212.415211] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 225.025329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 225.067584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.319309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 226.340906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 226.429724] Lustre: lustre-MDT0000: new disk, initializing [ 226.501864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 226.518898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 230.636335] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.209302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 243.315163] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 243.393165] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 243.401847] Lustre: Skipped 1 previous similar message [ 243.515707] Lustre: lustre-MDT0001: new disk, initializing [ 243.587647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.607700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 243.633600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 248.498631] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.439375] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 265.645963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 266.169966] Lustre: lustre-OST0000: new disk, initializing [ 266.190769] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 266.199875] Lustre: 8448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 266.306231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 268.811893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 268.830332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 269.028719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 274.708459] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 295.644883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 295.804877] Lustre: lustre-OST0001: new disk, initializing [ 295.808571] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 295.817548] Lustre: 9520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 295.872623] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 303.125831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 303.133567] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 303.268055] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 303.662680] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 318.266875] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 330.922807] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 341.030857] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing check_logdir /tmp/testlogs/ [ 348.121428] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing yml_node [ 353.617797] Lustre: DEBUG MARKER: Client: 2.17.57.2 [ 357.446471] Lustre: DEBUG MARKER: MDS: 2.17.57.2 [ 360.592547] Lustre: DEBUG MARKER: OSS: 2.17.57.2 [ 363.520797] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 17 02:16:08 EDT 2026 [ 382.659119] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 384.456138] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 402.400808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 402.409526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 402.436316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 405.474207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 405.476431] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 405.480859] Lustre: Skipped 2 previous similar messages [ 405.511061] Lustre: Skipped 2 previous similar messages [ 406.759928] Lustre: server umount lustre-MDT0000 complete [ 410.584966] LustreError: 7459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786947418 with bad export cookie 253341405454887709 [ 410.590348] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 410.593126] LustreError: 7459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 410.632473] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.653950] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 410.658132] Lustre: Skipped 2 previous similar messages [ 415.713619] LustreError: 10348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 415.713653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 415.752734] LustreError: 10348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 415.755058] Lustre: Skipped 1 previous similar message [ 417.189419] Lustre: server umount lustre-MDT0001 complete [ 421.886448] Lustre: server umount lustre-OST0000 complete [ 426.136045] Lustre: server umount lustre-OST0001 complete [ 434.373493] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 446.287623] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 490.875249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.290783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 491.353964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 491.464522] Lustre: lustre-MDT0000: new disk, initializing [ 491.580164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.599618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 497.887489] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 510.459418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.648893] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 510.707791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 510.712790] Lustre: Skipped 1 previous similar message [ 510.804589] Lustre: lustre-MDT0001: new disk, initializing [ 510.922268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 510.971496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 510.997245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 516.532713] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 526.962729] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 533.228203] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 541.104732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 541.564396] Lustre: lustre-OST0000: new disk, initializing [ 541.570579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 541.582745] Lustre: 17057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 541.755157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 542.648864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 542.668875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 542.870198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 550.246994] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 560.781858] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 565.778281] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 565.995167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 572.644616] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 572.917446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 578.531742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 578.543287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.548929] LustreError: Skipped 1 previous similar message [ 578.554891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 578.561591] Lustre: Skipped 2 previous similar messages [ 578.578391] Lustre: Skipped 1 previous similar message [ 588.794249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 588.802708] Lustre: Skipped 3 previous similar messages [ 589.279324] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 589.688932] Lustre: server umount lustre-OST0000 complete [ 593.903508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 608.223228] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 608.611875] Lustre: server umount lustre-MDT0000 complete [ 609.264863] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.302940] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 613.509082] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786947621 with bad export cookie 253341405454889242 [ 613.526503] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 613.537470] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 614.002890] Lustre: server umount lustre-MDT0001 complete [ 619.012544] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 02:20:24 (1786947624) [ 627.779323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.928707] LustreError: 19326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.088682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.356804] LustreError: 19327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.394188] LustreError: 19327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 634.750142] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 638.445229] LustreError: 19326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 645.615919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.290600] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 664.985185] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 673.524022] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 682.321316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 690.295910] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 700.797332] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 710.511788] Lustre: server umount lustre-OST0000 complete [ 717.795194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 717.804821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 717.831661] Lustre: Skipped 1 previous similar message [ 717.838039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 717.851267] Lustre: Skipped 6 previous similar messages [ 726.298254] Lustre: server umount lustre-MDT0000 complete [ 728.031082] Lustre: 3645:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786947719/real 1786947719] req@ffff8fddffd22680 x1873749899145728/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786947735 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 728.078933] LustreError: 19322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 728.107212] LustreError: 19322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 730.912737] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786947722/real 1786947722] req@ffff8fdded8f1f80 x1873749899146880/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786947738 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 731.053901] LustreError: 19306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786947738 with bad export cookie 253341405454890124 [ 731.059344] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 731.066654] LustreError: 19306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 731.461607] Lustre: server umount lustre-MDT0001 complete [ 745.162630] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 750.405591] Key type lgssc unregistered [ 750.903632] LNet: 22870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.916470] LNetError: 22870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 750.931608] LNet: Removed LNI 192.168.202.122@tcp [ 752.370596] Key type .llcrypt unregistered [ 752.373790] Key type ._llcrypt unregistered [ 783.015593] Key type ._llcrypt registered [ 783.023383] Key type .llcrypt registered [ 783.221136] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 784.496818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 784.615282] alg: No test for adler32 (adler32-zlib) [ 785.868709] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 786.103284] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 787.799198] Key type lgssc registered [ 788.977665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 796.683173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 796.699234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.233562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 803.419844] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 808.443903] LustreError: 24285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 808.476483] LustreError: 24285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 812.352306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.636373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 817.298575] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 824.294465] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 829.811505] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 837.875174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 838.061784] LustreError: 26106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 838.392723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 843.539921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 844.984575] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 853.501975] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 869.344280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 869.345991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 869.356419] LustreError: Skipped 1 previous similar message [ 869.358273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 869.406164] Lustre: Skipped 1 previous similar message [ 870.258888] Lustre: server umount lustre-OST0000 complete [ 874.474951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.489803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 874.492765] Lustre: Skipped 1 previous similar message [ 879.585824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 879.591087] Lustre: Skipped 1 previous similar message [ 880.541527] Lustre: server umount lustre-MDT0000 complete [ 884.021365] LustreError: 24265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786947891 with bad export cookie 5227829824139192936 [ 884.057209] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 884.426471] Lustre: server umount lustre-MDT0001 complete [ 899.100215] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 902.620258] Key type lgssc unregistered [ 903.129904] LNet: 28262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.141440] LNetError: 28262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.171406] LNet: Removed LNI 192.168.202.122@tcp [ 904.188748] Key type .llcrypt unregistered [ 904.191620] Key type ._llcrypt unregistered [ 930.941742] Key type ._llcrypt registered [ 930.943581] Key type .llcrypt registered [ 931.036611] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 932.206947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 932.300372] alg: No test for adler32 (adler32-zlib) [ 933.423842] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 933.668031] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 935.351356] Key type lgssc registered [ 936.546616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 944.898824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 944.921416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 946.487783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.606724] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 956.912754] LustreError: 29678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.946793] LustreError: 29678:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 961.431150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.813343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 966.803499] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 974.703741] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 979.966033] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 994.978382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 995.330159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.192902] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1010.100881] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1016.847109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1027.051690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.057729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1027.067250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1032.146269] Lustre: server umount lustre-OST0000 complete [ 1037.283833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.287952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1037.311336] Lustre: Skipped 2 previous similar messages [ 1037.322885] Lustre: Skipped 1 previous similar message [ 1042.155097] Lustre: server umount lustre-MDT0000 complete [ 1042.404526] LustreError: 29679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.414671] LustreError: 29679:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1046.135956] LustreError: 29659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948053 with bad export cookie 10605963411999753079 [ 1046.153453] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1046.625824] Lustre: server umount lustre-MDT0001 complete [ 1058.483938] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 1061.380444] Key type lgssc unregistered [ 1061.695660] LNet: 33585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.706240] LNetError: 33585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.759200] LNet: Removed LNI 192.168.202.122@tcp [ 1064.144269] Key type .llcrypt unregistered [ 1064.148651] Key type ._llcrypt unregistered [ 1092.103775] Key type ._llcrypt registered [ 1092.106667] Key type .llcrypt registered [ 1092.262427] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1093.648871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1093.675586] alg: No test for adler32 (adler32-zlib) [ 1094.829927] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 1095.160421] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 1096.903285] Key type lgssc registered [ 1098.210613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1106.040127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1106.067580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.575845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.497504] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1117.156241] LustreError: 34998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.173302] LustreError: 34998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1120.336800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.817270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1125.508706] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1134.040463] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1140.916759] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1148.804686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1149.209610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1152.314085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1155.047984] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1161.926067] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1178.085045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1178.090421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.116598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1181.158139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1181.172360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1181.194443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.828160] Lustre: server umount lustre-OST0000 complete [ 1187.307978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1187.316301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1187.341542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1187.350196] Lustre: Skipped 1 previous similar message [ 1191.392404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.395157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1191.437380] Lustre: Skipped 1 previous similar message [ 1192.792938] Lustre: server umount lustre-MDT0000 complete [ 1196.513728] LustreError: 34998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.524841] LustreError: 34998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1196.802881] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948204 with bad export cookie 9817923223436008226 [ 1196.824460] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1197.231556] Lustre: server umount lustre-MDT0001 complete [ 1210.335325] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 1212.907333] Key type lgssc unregistered [ 1213.216737] LNet: 38554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.230182] LNetError: 38554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.272982] LNet: Removed LNI 192.168.202.122@tcp [ 1214.455965] Key type .llcrypt unregistered [ 1214.461279] Key type ._llcrypt unregistered [ 1239.860971] Key type ._llcrypt registered [ 1239.865875] Key type .llcrypt registered [ 1240.027143] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1241.450379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1241.599424] alg: No test for adler32 (adler32-zlib) [ 1242.684526] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 1242.891761] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 1244.527339] Key type lgssc registered [ 1246.204272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1255.124427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1255.143898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1256.767817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1261.573474] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1266.148870] LustreError: 39967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.177564] LustreError: 39967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1269.876862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1270.183631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1275.333754] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1285.389373] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1291.599779] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1299.029140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1299.495680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1301.566098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1306.976388] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1316.334954] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1323.445830] Lustre: lustre-OST0000: Not available for connect from 192.168.202.22@tcp (stopping) [ 1325.024571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1325.033485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.080025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1325.092330] Lustre: Skipped 1 previous similar message [ 1328.521309] LustreError: 41789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.727058] Lustre: server umount lustre-OST0000 complete [ 1352.177546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1352.188374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.201160] Lustre: Skipped 1 previous similar message [ 1352.219179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1355.746593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1355.747477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.190922] Lustre: server umount lustre-MDT0000 complete [ 1360.867843] LustreError: 39968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.893231] LustreError: 39968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1360.913473] LustreError: 39949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948368 with bad export cookie 10490521921439754386 [ 1360.932921] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1367.427852] Lustre: server umount lustre-MDT0001 complete [ 1378.445945] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 1381.594296] Key type lgssc unregistered [ 1381.918851] LNet: 43680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1381.930710] LNetError: 43680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.951926] LNet: Removed LNI 192.168.202.122@tcp [ 1382.721485] Key type .llcrypt unregistered [ 1382.726905] Key type ._llcrypt unregistered [ 1406.655912] Key type ._llcrypt registered [ 1406.658498] Key type .llcrypt registered [ 1406.774359] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1408.346320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1408.361488] alg: No test for adler32 (adler32-zlib) [ 1409.400702] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 1409.716240] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 1411.343285] Key type lgssc registered [ 1412.379106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1420.537340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1420.561360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.972754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.024532] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1431.527665] LustreError: 45095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1431.556524] LustreError: 45095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1436.641057] LustreError: 45094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.909334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1437.353631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1442.025481] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1449.577319] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1456.602114] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1464.479578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1464.709310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1466.812698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1472.853150] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1480.638994] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1487.329778] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1487.332078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1487.350837] Lustre: Skipped 1 previous similar message [ 1487.371790] Lustre: Skipped 2 previous similar messages [ 1488.802368] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (stopping) [ 1489.428257] Lustre: server umount lustre-MDT0000 complete [ 1492.463285] LustreError: 45095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.491903] LustreError: 45095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1493.338684] LustreError: 46937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948501 with bad export cookie 11183553757296198966 [ 1493.344736] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1493.366803] LustreError: 46937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1493.934799] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.22@tcp (stopping) [ 1497.574604] LustreError: 45089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1497.578523] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.584100] LustreError: 45089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1497.602633] Lustre: Skipped 1 previous similar message [ 1497.610780] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1499.846652] Lustre: server umount lustre-MDT0001 complete [ 1528.799233] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1529.088762] Lustre: server umount lustre-OST0000 complete [ 1546.080434] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 1548.933963] Key type lgssc unregistered [ 1549.206901] LNet: 48959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.219942] LNetError: 48959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.235731] LNet: Removed LNI 192.168.202.122@tcp [ 1550.012394] Key type .llcrypt unregistered [ 1550.014448] Key type ._llcrypt unregistered [ 1574.907195] Key type ._llcrypt registered [ 1574.916825] Key type .llcrypt registered [ 1575.047270] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1576.268215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1576.293971] alg: No test for adler32 (adler32-zlib) [ 1577.479959] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 1577.784772] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 1579.495205] Key type lgssc registered [ 1580.555955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1588.342089] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1588.373353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1604.063518] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1609.183386] LustreError: 50346:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.122@tcp: failed processing log, type 4: rc = -110 [ 1636.000308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.904388] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1650.926478] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1818.155366] Lustre: server umount lustre-OST0000 complete [ 1835.101097] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 1838.051321] Key type lgssc unregistered [ 1838.414214] LNet: 52026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1838.427228] LNetError: 52026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1838.440431] LNet: Removed LNI 192.168.202.122@tcp [ 1839.157283] Key type .llcrypt unregistered [ 1839.165343] Key type ._llcrypt unregistered [ 1864.981903] Key type ._llcrypt registered [ 1864.984418] Key type .llcrypt registered [ 1865.097796] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1866.226045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1866.352786] alg: No test for adler32 (adler32-zlib) [ 1867.480273] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 1867.755852] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 1869.415181] Key type lgssc registered [ 1870.831726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1878.813891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1878.853887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.500865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1885.603930] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1890.795464] LustreError: 53443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1890.812524] LustreError: 53443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1894.754803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1895.099693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1899.808340] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1908.597769] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1915.468564] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1924.472808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1925.128655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1926.235803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:257) [ 1935.541569] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 1945.046667] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1949.666164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.671636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1949.693830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1949.737519] Lustre: Skipped 1 previous similar message [ 1953.317501] Lustre: server umount lustre-OST0000 complete [ 1959.905668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1959.921120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1959.939437] Lustre: Skipped 2 previous similar messages [ 1959.953044] Lustre: Skipped 1 previous similar message [ 1964.257215] Lustre: server umount lustre-MDT0000 complete [ 1965.025653] LustreError: 53444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1965.046909] LustreError: 53444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1968.195763] LustreError: 53422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948976 with bad export cookie 3042495005175184089 [ 1968.215994] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1968.670727] Lustre: server umount lustre-MDT0001 complete [ 1980.676842] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 1983.340487] Key type lgssc unregistered [ 1983.717422] LNet: 56977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1983.739168] LNetError: 56977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.754872] LNet: Removed LNI 192.168.202.122@tcp [ 1984.429344] Key type .llcrypt unregistered [ 1984.431488] Key type ._llcrypt unregistered [ 2012.669821] Key type ._llcrypt registered [ 2012.673208] Key type .llcrypt registered [ 2012.781948] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2014.022109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2014.094777] alg: No test for adler32 (adler32-zlib) [ 2015.150590] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 2015.404518] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2017.111229] Key type lgssc registered [ 2018.224613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2025.897829] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2025.914779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2041.505268] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2046.624163] LustreError: 58366:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.122@tcp: failed processing log, type 4: rc = -110 [ 2073.311231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.723721] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2091.362412] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2099.473438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2099.842471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2099.939792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:289) [ 2104.061829] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2105.313587] LustreError: 58391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.346770] LustreError: 58391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2107.363450] LustreError: 58392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.392725] LustreError: 58392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2112.482971] LustreError: 59400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.108580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2113.545530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2118.619808] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2126.819899] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2132.452356] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2136.049559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2136.061240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.082880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2139.107716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2139.108251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2139.133328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.736665] Lustre: server umount lustre-OST0000 complete [ 2169.835834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.838040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.850219] Lustre: Skipped 1 previous similar message [ 2169.852762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2169.888505] Lustre: Skipped 2 previous similar messages [ 2172.724197] Lustre: server umount lustre-MDT0000 complete [ 2174.948536] LustreError: 59400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2174.973218] LustreError: 59400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2176.462633] LustreError: 58373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949184 with bad export cookie 11522750836906853445 [ 2176.465669] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2176.476016] LustreError: 58373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2182.975640] Lustre: server umount lustre-MDT0001 complete [ 2197.532515] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2201.074269] Key type lgssc unregistered [ 2201.438975] LNet: 62115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2201.446366] LNetError: 62115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2201.458889] LNet: Removed LNI 192.168.202.122@tcp [ 2202.452987] Key type .llcrypt unregistered [ 2202.455468] Key type ._llcrypt unregistered [ 2226.978829] Key type ._llcrypt registered [ 2226.981732] Key type .llcrypt registered [ 2227.099539] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2228.153135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2228.408249] alg: No test for adler32 (adler32-zlib) [ 2229.486405] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 2229.760774] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2231.448267] Key type lgssc registered [ 2232.830717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2242.002691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2242.039647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.563230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.883861] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2253.794456] LustreError: 63529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.810083] LustreError: 63529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2258.924315] LustreError: 63528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.055828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.575147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2265.569172] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2273.858543] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2280.428963] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2286.982504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2287.473066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.548063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:321) [ 2294.584735] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2303.064966] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2332.913298] Lustre: server umount lustre-OST0000 complete [ 2340.836280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.843658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2340.851946] Lustre: Skipped 1 previous similar message [ 2340.876022] Lustre: Skipped 1 previous similar message [ 2345.969256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2345.975186] Lustre: Skipped 1 previous similar message [ 2347.067508] Lustre: server umount lustre-MDT0000 complete [ 2350.759335] LustreError: 63510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949358 with bad export cookie 5157807225475691898 [ 2350.760591] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2350.766793] LustreError: 63510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2360.447661] Lustre: server umount lustre-MDT0001 complete [ 2370.662472] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2373.632196] Key type lgssc unregistered [ 2373.892614] LNet: 67092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2373.899709] LNetError: 67092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2373.921049] LNet: Removed LNI 192.168.202.122@tcp [ 2374.729184] Key type .llcrypt unregistered [ 2374.732819] Key type ._llcrypt unregistered [ 2399.261657] Key type ._llcrypt registered [ 2399.270434] Key type .llcrypt registered [ 2399.414343] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2400.697476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2400.739880] alg: No test for adler32 (adler32-zlib) [ 2401.718926] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 2401.934859] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2403.631397] Key type lgssc registered [ 2404.770914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2412.870720] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 02:50:18 (1786949418) [ 2419.765982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2419.792403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.136099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2424.882803] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2430.433386] LustreError: 68778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.440747] LustreError: 68778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2433.013745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.305546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2438.169535] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2445.692369] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2450.325550] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2456.537676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2456.871411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2462.002435] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2466.110725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:353) [ 2468.533698] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2474.761506] Lustre: Failing over lustre-MDT0000 [ 2475.044945] Lustre: server umount lustre-MDT0000 complete [ 2476.472320] LustreError: 68773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.520455] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.530116] Lustre: Skipped 1 previous similar message [ 2481.558553] LustreError: 68773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.574186] LustreError: 68773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2483.483775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2483.597197] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2483.721964] Lustre: *** cfs_fail_loc=135, val=0*** [ 2483.790952] LustreError: 71865:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2483.796349] LustreError: 71865:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.122@tcp: cfg command failed: rc = -2 [ 2483.803680] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2483.803680] [ 2483.809421] LustreError: MGC192.168.202.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2483.817115] LustreError: 71854:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2483.827310] LustreError: 71854:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2483.836575] LustreError: 71854:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2483.841836] LustreError: 71854:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2483.862234] Lustre: server umount lustre-MDT0000 complete [ 2483.865393] LustreError: 71854:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2486.671397] LustreError: 71270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.696807] LustreError: 71270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2489.781939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2489.904612] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2489.918335] LustreError: 72210:0:(ldlm_resource.c:1208:ldlm_resource_complain()) MGC192.168.202.122@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000066ff055b) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2489.944088] LustreError: 68770:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2490.209392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2490.259727] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2491.807708] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2494.850785] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2495.466412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2495.502870] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2495.530335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:385) [ 2501.088372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2501.099057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2501.113051] Lustre: Skipped 1 previous similar message [ 2501.116987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2502.976340] Lustre: server umount lustre-OST0000 complete [ 2510.817085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.824906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2510.836941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2510.848497] Lustre: Skipped 1 previous similar message [ 2512.869365] Lustre: server umount lustre-MDT0000 complete [ 2515.937816] LustreError: 71270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.955477] LustreError: 71270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2516.536044] LustreError: 68759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949524 with bad export cookie 18276432657716771252 [ 2516.544744] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2516.549756] LustreError: 68759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2526.833872] Lustre: server umount lustre-MDT0001 complete [ 2536.122301] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2538.240309] Key type lgssc unregistered [ 2538.450519] LNet: 73647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2538.456879] LNetError: 73647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2539.496436] LNet: Removed LNI 192.168.202.122@tcp [ 2540.130661] Key type .llcrypt unregistered [ 2540.134375] Key type ._llcrypt unregistered [ 2558.565534] Key type ._llcrypt registered [ 2558.568971] Key type .llcrypt registered [ 2558.662244] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2559.579308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2559.701906] alg: No test for adler32 (adler32-zlib) [ 2560.703689] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 2560.919587] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2562.559219] Key type lgssc registered [ 2563.596790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2570.203901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2570.217867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.608533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2574.999783] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2581.984404] LustreError: 75040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.996539] LustreError: 75040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2582.007840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.279922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2586.269321] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2591.928892] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2596.921870] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2602.951198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2603.229343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.498749] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2615.504064] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2616.573436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:417) [ 2621.652712] Lustre: Failing over lustre-MDT0000 [ 2621.923464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.926372] LustreError: 76878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.937825] Lustre: Skipped 2 previous similar messages [ 2621.955105] LustreError: 76878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2621.982781] Lustre: server umount lustre-MDT0000 complete [ 2623.401209] LustreError: 75034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.053396] LustreError: 75040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.070847] LustreError: 75040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2629.806911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.916484] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2630.047226] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2630.155508] LustreError: 78125:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2630.161841] LustreError: 78125:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.122@tcp: cfg command failed: rc = -9 [ 2630.166873] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2630.166873] [ 2630.174353] LustreError: MGC192.168.202.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2630.182479] LustreError: 78114:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2630.187343] LustreError: 78114:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2630.200391] LustreError: 78114:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2630.205268] LustreError: 78114:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2630.225410] Lustre: server umount lustre-MDT0000 complete [ 2630.227904] LustreError: 78114:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2632.163340] LustreError: 75036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.172207] LustreError: 75036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2635.705899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.831503] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2636.013447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2636.073575] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2638.730580] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2639.714903] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2641.388531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2641.420809] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2641.453378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:449) [ 2646.501372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2646.508971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2646.509558] Lustre: Skipped 1 previous similar message [ 2646.522462] Lustre: Skipped 1 previous similar message [ 2648.312406] Lustre: server umount lustre-OST0000 complete [ 2651.617438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.632464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2651.633599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2656.739672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2656.747277] Lustre: Skipped 2 previous similar messages [ 2657.583352] Lustre: server umount lustre-MDT0000 complete [ 2660.849781] LustreError: 75021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949668 with bad export cookie 4952859349823543708 [ 2660.851094] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2660.858113] LustreError: 75021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2661.145383] Lustre: server umount lustre-MDT0001 complete [ 2669.909836] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2672.112781] Key type lgssc unregistered [ 2672.376589] LNet: 79910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2672.381425] LNetError: 79910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2672.402830] LNet: Removed LNI 192.168.202.122@tcp [ 2673.304510] Key type .llcrypt unregistered [ 2673.310714] Key type ._llcrypt unregistered [ 2692.018240] Key type ._llcrypt registered [ 2692.025504] Key type .llcrypt registered [ 2692.252235] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2692.995307] Key type .llcrypt unregistered [ 2693.002544] Key type ._llcrypt unregistered [ 2728.800653] Key type ._llcrypt registered [ 2728.801821] Key type .llcrypt registered [ 2728.976186] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2731.058596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2731.098945] alg: No test for adler32 (adler32-zlib) [ 2732.213878] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 2732.542835] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2734.271161] Key type lgssc registered [ 2735.542169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2743.971901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2744.013557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2745.697227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2750.500077] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2756.068605] LustreError: 81820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.099860] LustreError: 81820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2760.826114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.188408] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2761.477872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2766.403830] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2774.223428] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2780.339196] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2787.950210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2788.548928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2790.644111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:481) [ 2795.807593] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2804.718792] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2816.480298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2816.496514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2816.515194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2817.505057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2817.513744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2817.528495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2817.536073] Lustre: Skipped 1 previous similar message [ 2820.419698] Lustre: server umount lustre-OST0000 complete [ 2827.745667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2827.750739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2827.782932] Lustre: Skipped 1 previous similar message [ 2827.797391] Lustre: Skipped 1 previous similar message [ 2830.957944] Lustre: server umount lustre-MDT0000 complete [ 2832.866388] LustreError: 81821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2832.892756] LustreError: 81821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2834.913064] LustreError: 81800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949842 with bad export cookie 3974351941305276512 [ 2834.920927] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2834.924827] LustreError: 81800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2835.259844] Lustre: server umount lustre-MDT0001 complete [ 2847.605391] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2851.049787] Key type lgssc unregistered [ 2851.419370] LNet: 85355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2851.427411] LNetError: 85355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2851.472230] LNet: Removed LNI 192.168.202.122@tcp [ 2852.512291] Key type .llcrypt unregistered [ 2852.515536] Key type ._llcrypt unregistered [ 2879.638607] Key type ._llcrypt registered [ 2879.641553] Key type .llcrypt registered [ 2879.744823] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2880.960227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2880.985046] alg: No test for adler32 (adler32-zlib) [ 2882.055287] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 2882.251085] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2883.887615] Key type lgssc registered [ 2885.205590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2892.850681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2892.876581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.514082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.726919] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2905.057103] LustreError: 86770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.071908] LustreError: 86770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2908.226349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2908.702286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2912.896306] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2919.537990] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2926.289736] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2933.035800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2933.157717] LustreError: 88590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.368589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.793544] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 2939.905136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:483 to 0x280000401:513) [ 2947.722681] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2957.754328] Lustre: server umount lustre-OST0000 complete [ 2964.962136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2964.972815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.016785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2965.472553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2970.603212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2970.608558] Lustre: Skipped 3 previous similar messages [ 2975.713150] Lustre: 86082:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949967/real 1786949967] req@00000000f23059df x1873752767883392/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786949983 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2975.713845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2975.756235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2975.776321] Lustre: Skipped 1 previous similar message [ 2976.223162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2976.799170] Lustre: 86082:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949968/real 1786949968] req@00000000c81840a9 x1873752767884416/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786949984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2978.526236] Lustre: server umount lustre-MDT0000 complete [ 2980.836497] LustreError: 86769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2980.860348] LustreError: 86769:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2981.855541] Lustre: 86080:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949973/real 1786949973] req@00000000f2aa209c x1873752767886336/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786949989 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2982.859124] LustreError: 86749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949990 with bad export cookie 5217271489665929837 [ 2982.867366] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2982.873241] LustreError: 86749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2983.706779] Lustre: server umount lustre-MDT0001 complete [ 2998.212968] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 3001.611152] Key type lgssc unregistered [ 3001.991555] LNet: 90305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3001.995889] LNetError: 90305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3002.010208] LNet: Removed LNI 192.168.202.122@tcp [ 3003.159110] Key type .llcrypt unregistered [ 3003.160807] Key type ._llcrypt unregistered [ 3031.417564] Key type ._llcrypt registered [ 3031.420813] Key type .llcrypt registered [ 3031.570900] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3033.021138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3033.061672] alg: No test for adler32 (adler32-zlib) [ 3034.282543] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 3034.631752] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 3036.303441] Key type lgssc registered [ 3037.547387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3046.427884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3046.450373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.006253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3053.711089] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3058.148878] LustreError: 91720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3058.157054] LustreError: 91720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3063.269021] LustreError: 91721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3063.846866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3064.393865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3069.228728] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3077.242880] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3082.867294] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3091.340474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3091.712695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.738553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:483 to 0x280000401:545) [ 3098.197552] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3106.779721] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3123.172777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3123.188587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.207756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3123.597358] Lustre: server umount lustre-OST0000 complete [ 3130.858136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.858782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3130.901303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.913621] Lustre: Skipped 1 previous similar message [ 3133.656110] Lustre: server umount lustre-MDT0000 complete [ 3135.973697] LustreError: 91716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.017414] LustreError: 91716:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3137.566883] LustreError: 92470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786950145 with bad export cookie 13076898966422667636 [ 3137.576950] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3137.593239] LustreError: 92470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3147.957586] Lustre: server umount lustre-MDT0001 complete [ 3159.831615] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 3163.175514] Key type lgssc unregistered [ 3163.464561] LNet: 95295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3163.484483] LNetError: 95295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3163.513371] LNet: Removed LNI 192.168.202.122@tcp [ 3164.553246] Key type .llcrypt unregistered [ 3164.555782] Key type ._llcrypt unregistered [ 3192.376289] Key type ._llcrypt registered [ 3192.378873] Key type .llcrypt registered [ 3192.484826] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3193.969268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3194.215933] alg: No test for adler32 (adler32-zlib) [ 3195.599579] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 3196.032646] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 3197.791730] Key type lgssc registered [ 3199.179198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3208.120334] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3208.132967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3223.647467] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3228.769745] LustreError: 96681:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.122@tcp: failed processing log, type 4: rc = -110 [ 3255.455952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3262.270757] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3273.629766] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3279.389924] Lustre: server umount lustre-OST0000 complete [ 3288.049846] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 03:04:53 (1786950293) [ 3297.312646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.931295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3303.388893] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3303.391760] LustreError: 98319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.437454] LustreError: 98319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3307.496192] LustreError: 98320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.577651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3312.614918] LustreError: 98319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3313.166983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3319.396347] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3329.863167] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3339.045145] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3345.633184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3346.246827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3353.410581] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3361.395874] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3367.794517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:548 to 0x280000401:577) [ 3383.269171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3383.269301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3383.275045] LustreError: Skipped 1 previous similar message [ 3383.282085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3383.288148] Lustre: Skipped 1 previous similar message [ 3388.397801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3388.418088] Lustre: Skipped 1 previous similar message [ 3389.434165] Lustre: server umount lustre-OST0000 complete [ 3395.040613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3395.053978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.073091] Lustre: Skipped 1 previous similar message [ 3395.083567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3398.636908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3398.651981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.745096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3403.757016] Lustre: Skipped 2 previous similar messages [ 3408.351143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3408.793168] Lustre: server umount lustre-MDT0000 complete [ 3408.867321] LustreError: 99047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3408.892532] LustreError: 99047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3412.594567] LustreError: 98302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786950420 with bad export cookie 7479932982287954859 [ 3412.601293] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3412.615080] LustreError: 98302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3413.019453] Lustre: server umount lustre-MDT0001 complete [ 3427.324646] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 3431.704650] Key type lgssc unregistered [ 3432.203517] LNet: 102043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3432.212625] LNetError: 102043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3432.228850] LNet: Removed LNI 192.168.202.122@tcp [ 3433.125676] Key type .llcrypt unregistered [ 3433.127869] Key type ._llcrypt unregistered [ 3486.578913] Key type ._llcrypt registered [ 3486.583511] Key type .llcrypt registered [ 3486.702849] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3488.100205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3488.342023] alg: No test for adler32 (adler32-zlib) [ 3489.636844] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 3489.792437] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 3491.424938] Key type lgssc registered [ 3492.230744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3501.590979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3501.612269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3503.246159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.537423] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3513.827635] LustreError: 104233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3513.867814] LustreError: 104233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3518.293042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3518.722643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3523.002691] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3531.675537] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3537.794134] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3545.186563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3545.726529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3548.805421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:548 to 0x280000401:609) [ 3552.766765] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3562.869940] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3574.758328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3574.770836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3574.785717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3578.848186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3578.865028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3578.876447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3578.895037] Lustre: Skipped 1 previous similar message [ 3580.417694] Lustre: server umount lustre-OST0000 complete [ 3585.511938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3585.524482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3585.559129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3589.088922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3589.110624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.214747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3594.224452] Lustre: Skipped 2 previous similar messages [ 3599.328332] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3599.835925] Lustre: server umount lustre-MDT0000 complete [ 3603.434437] LustreError: 104233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3603.456181] LustreError: 104233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3604.905923] LustreError: 104213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786950612 with bad export cookie 17978498946180702365 [ 3604.909638] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3604.932486] LustreError: 104213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3605.585154] Lustre: server umount lustre-MDT0001 complete [ 3619.411617] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 3623.750711] Key type lgssc unregistered [ 3624.073614] LNet: 107774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3624.084809] LNetError: 107774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3624.124486] LNet: Removed LNI 192.168.202.122@tcp [ 3625.244069] Key type .llcrypt unregistered [ 3625.246938] Key type ._llcrypt unregistered [ 3648.378396] Key type ._llcrypt registered [ 3648.384768] Key type .llcrypt registered [ 3648.490916] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3649.647581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3649.739774] alg: No test for adler32 (adler32-zlib) [ 3650.851526] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 3651.126990] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 3652.783304] Key type lgssc registered [ 3653.786184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3662.522319] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3662.538240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3678.303525] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3683.423401] LustreError: 109018:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.122@tcp: failed processing log, type 4: rc = -110 [ 3710.177860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3718.041676] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3729.302452] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3738.300223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3738.875270] LustreError: MGC192.168.202.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3738.915397] LustreError: 110043:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3738.940466] LustreError: 110043:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3738.968872] LustreError: 110043:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3739.182919] Lustre: server umount lustre-MDT0000 complete [ 3739.191713] LustreError: 110043:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3760.095200] Lustre: 108356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786950751/real 1786950751] req@00000000e1ff90ca x1873753573166464/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786950767 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3760.147474] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3760.175801] LustreError: 109030:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3760.458914] Lustre: server umount lustre-OST0000 complete [ 3771.248252] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 3786.690213] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3829.279378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.582439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3829.603927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3829.687086] Lustre: lustre-MDT0000: new disk, initializing [ 3829.837740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.867848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3835.755988] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3849.546413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3849.739412] Lustre: 112858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3849.831725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3849.837129] Lustre: Skipped 1 previous similar message [ 3849.983434] Lustre: lustre-MDT0001: new disk, initializing [ 3850.189856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3850.252240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3850.269698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3855.352641] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3865.918291] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3873.028544] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3881.476115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3881.778121] Lustre: lustre-OST0000: new disk, initializing [ 3881.782977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3881.787225] Lustre: 114832:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3881.872141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3887.648312] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 3887.682487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3887.695286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3887.791980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3899.594459] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3906.551345] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3906.825378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3912.780188] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3913.044363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3918.816920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3918.827133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3918.832870] LustreError: Skipped 1 previous similar message [ 3918.833634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3918.838955] Lustre: Skipped 1 previous similar message [ 3918.854626] Lustre: Skipped 1 previous similar message [ 3921.974307] Lustre: server umount lustre-OST0000 complete [ 3927.013204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3927.032684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3927.053825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3929.058574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3929.059436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3929.075080] Lustre: Skipped 1 previous similar message [ 3933.087728] Lustre: server umount lustre-MDT0000 complete [ 3937.100116] LustreError: 112851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786950944 with bad export cookie 13210853795250552182 [ 3937.119241] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3937.122467] LustreError: 112851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3937.423187] Lustre: server umount lustre-MDT0001 complete [ 3946.447440] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 03:15:52 (1786950952) [ 3949.709584] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3970.116256] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 3981.318816] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 4022.517245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4022.791696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4022.835873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4022.953702] Lustre: lustre-MDT0000: new disk, initializing [ 4023.043960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.082063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4027.604689] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4040.963166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4041.180811] Lustre: 119734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4041.259602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4041.264588] Lustre: Skipped 1 previous similar message [ 4041.397441] Lustre: lustre-MDT0001: new disk, initializing [ 4041.525600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4041.565269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4041.581274] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4046.980301] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4058.295783] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4065.572917] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4072.370984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4072.610880] Lustre: lustre-OST0000: new disk, initializing [ 4072.614458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4072.620313] Lustre: 121707:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4072.703788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4080.598352] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4080.753534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4080.785306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4080.985079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4091.017389] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4097.706945] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4097.968162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4104.608899] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4104.914806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4111.845336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4111.846900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4111.855262] LustreError: Skipped 1 previous similar message [ 4111.869538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4111.875380] Lustre: Skipped 1 previous similar message [ 4111.899457] Lustre: Skipped 1 previous similar message [ 4113.514568] Lustre: server umount lustre-OST0000 complete [ 4118.499089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4118.517562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4118.552352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4124.249403] Lustre: server umount lustre-MDT0000 complete [ 4127.202268] LustreError: 119747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4127.215256] LustreError: 119747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4128.064313] LustreError: 119728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951135 with bad export cookie 13210853795250553064 [ 4128.071822] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4128.078675] LustreError: 119728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4128.424224] Lustre: server umount lustre-MDT0001 complete [ 4137.389374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4137.788485] LustreError: 123934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.856067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4142.555339] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4143.073397] LustreError: 123935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.175641] LustreError: 123934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4150.810465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4155.514823] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4164.759612] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4173.238547] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4181.637642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4188.904484] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4196.620357] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4203.470736] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4208.608572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4208.619786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.635238] Lustre: Skipped 1 previous similar message [ 4208.646422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4208.652440] Lustre: Skipped 2 previous similar messages [ 4214.455676] Lustre: server umount lustre-OST0000 complete [ 4227.040886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4227.053667] Lustre: Skipped 5 previous similar messages [ 4227.324787] Lustre: server umount lustre-MDT0000 complete [ 4231.607804] LustreError: 123915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951239 with bad export cookie 13210853795250553946 [ 4231.613542] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4231.632930] LustreError: 123915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4232.040825] Lustre: server umount lustre-MDT0001 complete [ 4244.953890] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 4248.524787] Key type lgssc unregistered [ 4248.788761] LNet: 127566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4248.793665] LNetError: 127566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4248.826541] LNet: Removed LNI 192.168.202.122@tcp [ 4249.972192] Key type .llcrypt unregistered [ 4249.976485] Key type ._llcrypt unregistered [ 4274.018670] Key type ._llcrypt registered [ 4274.023809] Key type .llcrypt registered [ 4274.176440] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 4292.177403] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 4293.085431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4293.128238] alg: No test for adler32 (adler32-zlib) [ 4294.215551] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 4294.529210] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 4296.319216] Key type lgssc registered [ 4297.418379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4330.551683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4330.599334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4331.913440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4331.962953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4332.113452] Lustre: lustre-MDT0000: new disk, initializing [ 4332.213147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4332.232796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4337.152518] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4348.888800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4349.006701] Lustre: 130727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4349.052980] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4349.058671] Lustre: Skipped 1 previous similar message [ 4349.140490] Lustre: lustre-MDT0001: new disk, initializing [ 4349.295343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4349.363044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4349.379824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4355.529427] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4366.006333] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4372.322861] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4379.811347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4380.220437] Lustre: lustre-OST0000: new disk, initializing [ 4380.242593] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4380.268321] Lustre: 132694:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4380.399355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4386.529816] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4388.444931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4388.461383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4388.591119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4395.804758] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4400.320652] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4400.521412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4406.964303] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4407.271752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4409.312528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4409.315730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4409.336206] LustreError: Skipped 1 previous similar message [ 4409.341901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4409.358398] Lustre: Skipped 1 previous similar message [ 4409.375724] Lustre: Skipped 1 previous similar message [ 4414.452522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4414.467413] Lustre: Skipped 1 previous similar message [ 4415.697452] Lustre: server umount lustre-OST0000 complete [ 4421.089580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4421.099901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4421.121351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4424.674197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4424.674605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4426.444664] Lustre: server umount lustre-MDT0000 complete [ 4430.555404] LustreError: 130718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951438 with bad export cookie 6911720422264266740 [ 4430.572120] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4430.572291] LustreError: 130718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4430.965435] Lustre: server umount lustre-MDT0001 complete [ 4441.134664] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 03:24:07 (1786951447) [ 4449.540339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4450.237834] LustreError: 135105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.256526] LustreError: 135105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4450.341750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4454.755516] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4455.395589] LustreError: 135106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4460.516663] LustreError: 135105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4462.833571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4463.152964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4469.551689] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4478.861251] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4485.560989] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4487.504893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4487.526132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4487.536831] Lustre: Skipped 2 previous similar messages [ 4493.800686] LustreError: 135844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4493.831824] LustreError: 135844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4494.029423] Lustre: server umount lustre-MDT0000 complete [ 4497.685512] LustreError: 135085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951505 with bad export cookie 6911720422264267622 [ 4497.694347] LustreError: 135085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4497.707583] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4498.200354] Lustre: server umount lustre-MDT0001 complete [ 4506.704224] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 03:25:12 (1786951512) [ 4513.894906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4529.569370] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4534.687594] LustreError: 137519:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.122@tcp: failed processing log, type 4: rc = -110 [ 4560.480609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.006354] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4575.367996] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4577.804372] Lustre: server umount lustre-OST0000 complete [ 4586.777687] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 03:26:32 (1786951592) [ 4595.697862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4596.144329] LustreError: 138967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4596.219664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4601.488580] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4606.448534] LustreError: 138968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4606.478616] LustreError: 138968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4612.573330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.375807] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4626.122732] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4632.412691] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4639.780288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4640.395765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4640.404723] Lustre: Skipped 1 previous similar message [ 4648.397379] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4657.169342] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4668.385698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4668.404939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4668.405692] Lustre: Skipped 2 previous similar messages [ 4668.418070] Lustre: Skipped 4 previous similar messages [ 4672.228515] Lustre: server umount lustre-MDT0000 complete [ 4673.511762] LustreError: 138967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4673.540425] LustreError: 138967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4676.344556] LustreError: 139704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951684 with bad export cookie 6911720422264268287 [ 4676.346594] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4676.368456] LustreError: 139704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4676.866629] Lustre: server umount lustre-MDT0001 complete [ 4690.505246] Lustre: server umount lustre-OST0000 complete [ 4701.855766] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 03:28:27 (1786951707) [ 4709.507587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4709.890820] LustreError: 142625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.028362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4715.255985] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4725.052489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4731.790760] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4740.544293] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4749.880994] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4759.088407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4770.331489] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4779.268799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4781.071899] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4786.875111] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4787.177357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4791.901703] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4792.203237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4794.851105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4794.853914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4794.874558] Lustre: Skipped 1 previous similar message [ 4794.879702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4800.696639] Lustre: server umount lustre-OST0000 complete [ 4807.137078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4812.221176] Lustre: server umount lustre-MDT0000 complete [ 4815.330952] LustreError: 143353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4815.362136] LustreError: 143353:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4816.614275] LustreError: 144470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951824 with bad export cookie 6911720422264269358 [ 4816.619401] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4816.624349] LustreError: 144470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4817.107310] Lustre: server umount lustre-MDT0001 complete [ 4827.715547] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 03:30:33 (1786951833) [ 4835.937579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4851.487643] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4856.607500] LustreError: 146786:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.122@tcp: failed processing log, type 4: rc = -110 [ 4882.335301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4882.345812] Lustre: Skipped 2 previous similar messages [ 4890.176646] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4899.207230] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4908.044211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4908.708079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4914.000645] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4923.863804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4929.971081] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 4939.716308] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4946.951182] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4954.140478] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4954.558584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4961.787941] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4962.187540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4965.859172] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4965.861229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4965.865598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4965.865607] Lustre: Skipped 6 previous similar messages [ 4965.896933] Lustre: Skipped 4 previous similar messages [ 4971.406872] Lustre: server umount lustre-MDT0000 complete [ 4976.096505] LustreError: 146811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.119927] LustreError: 146811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4976.272725] LustreError: 149937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951984 with bad export cookie 6911720422264270380 [ 4976.283807] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4976.290819] LustreError: 149937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4976.614924] Lustre: server umount lustre-MDT0001 complete [ 4991.488661] Lustre: server umount lustre-OST0000 complete [ 5003.973320] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 03:33:29 (1786952009) [ 5011.096035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5026.783694] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5031.904096] LustreError: 150985:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.122@tcp: failed processing log, type 4: rc = -110 [ 5065.293795] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5074.702411] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5082.525885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5083.268405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5089.523465] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5098.689780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5104.589591] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5112.575744] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5118.624819] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5126.782809] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5127.038573] Lustre: lustre-OST0001: new disk, initializing [ 5127.043083] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5127.055186] Lustre: 153858:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5135.628943] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5137.472384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5137.483905] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5137.608723] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5147.987900] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5153.252047] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5153.581358] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5157.981874] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5158.215993] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5160.423489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5160.439952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5160.476788] Lustre: Skipped 1 previous similar message [ 5160.487724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5160.507187] Lustre: Skipped 5 previous similar messages [ 5166.816716] Lustre: server umount lustre-OST0000 complete [ 5177.727796] Lustre: server umount lustre-OST0001 complete [ 5196.255350] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5196.484783] Lustre: server umount lustre-MDT0000 complete [ 5200.260378] LustreError: 150991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786952208 with bad export cookie 6911720422264271332 [ 5200.271213] LustreError: 150991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5200.275916] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5200.551926] Lustre: server umount lustre-MDT0001 complete [ 5226.787139] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 03:37:12 (1786952232) [ 5228.376897] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5230.375490] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5232.196222] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 03:37:18 (1786952238) [ 5239.231968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5239.435549] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5239.462285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5239.781457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5239.784593] Lustre: Skipped 6 previous similar messages [ 5244.970585] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5254.449718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5254.641766] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5254.719118] Lustre: 157423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5260.319365] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5269.007609] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5276.767979] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5284.294326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5284.463710] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5291.957287] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5292.177738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5300.479506] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5305.870779] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5306.275754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5312.098697] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5312.487488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5317.602210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5317.611950] LustreError: Skipped 2 previous similar messages [ 5317.616837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5317.628983] Lustre: Skipped 5 previous similar messages [ 5317.631936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5317.638035] Lustre: Skipped 10 previous similar messages [ 5329.375190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5329.810711] Lustre: server umount lustre-OST0000 complete [ 5356.307314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5358.891152] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.908271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.908806] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5358.925077] Lustre: Skipped 1 previous similar message [ 5364.891174] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5374.300353] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5379.128159] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5379.488939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5385.568638] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5385.847987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5392.273668] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9344d0352000.ost_server_uuid 50 [ 5393.885297] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9344d0352000.ost_server_uuid in FULL state after 0 sec [ 5394.070988] LustreError: 157430:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.22@tcp) returned error from blocking AST (req@00000000de1f6080 x1873754248727040 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000615a9bb2/0x5feb5d1aabdffb3e lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.22@tcp remote: 0x5a119413e49bf289 expref: 5 pid: 157429 timeout: 5494 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5394.128871] LustreError: lustre-MDT0000: A client on nid 192.168.202.22@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5394.137302] LustreError: 157419:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.22@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000615a9bb2/0x5feb5d1aabdffb3e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.22@tcp remote: 0x5a119413e49bf289 expref: 6 pid: 157429 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5400.034593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5400.059189] LustreError: Skipped 2 previous similar messages [ 5405.572163] Lustre: server umount lustre-OST0000 complete [ 5415.958344] Lustre: server umount lustre-MDT0000 complete [ 5420.271941] LustreError: 157416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786952428 with bad export cookie 6911720422264272515 [ 5420.286095] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5420.286685] LustreError: 157416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5427.124160] Lustre: server umount lustre-MDT0001 complete [ 5440.215472] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5443.803418] Key type lgssc unregistered [ 5444.187296] LNet: 163545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5444.204218] LNetError: 163545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5444.232727] LNet: Removed LNI 192.168.202.122@tcp [ 5445.288743] Key type .llcrypt unregistered [ 5445.291944] Key type ._llcrypt unregistered [ 5469.927494] Key type ._llcrypt registered [ 5469.935944] Key type .llcrypt registered [ 5470.164023] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5471.209085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5471.493170] alg: No test for adler32 (adler32-zlib) [ 5472.683631] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 5472.924618] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 5474.815225] Key type lgssc registered [ 5476.257663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5486.566469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5486.598574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5488.555840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5494.772499] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5498.855108] LustreError: 164959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5498.873853] LustreError: 164959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5503.970767] LustreError: 164958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5504.324205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5504.776319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5510.650298] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5519.427120] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5525.695692] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5532.683315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5532.963445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5534.016995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5539.530599] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5546.474505] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5556.726761] Lustre: Failing over lustre-MDT0000 [ 5557.066808] Lustre: server umount lustre-MDT0000 complete [ 5559.778327] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.782901] LustreError: 164954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5559.800031] Lustre: Skipped 1 previous similar message [ 5559.825023] LustreError: 164954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5564.898467] LustreError: 164959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5564.921088] LustreError: 164959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5569.494259] LustreError: 164940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.22@tcp arrived at 1786952577 with bad export cookie 8088691104191680027 [ 5570.019189] LustreError: 164955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5570.056598] LustreError: 164955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5576.161439] Lustre: 164268:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786952567/real 1786952567] req@00000000f7bf3500 x1873755483697024/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786952583 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5576.205447] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5577.383504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.280735] LustreError: 164953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5581.307300] LustreError: 164953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5586.611675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5586.645468] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5591.022656] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5592.037083] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5592.048659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5593.278213] Lustre: lustre-MDT0000: Denying connection for new client 61eb4a32-c9ef-43fb-ba9f-3d2e208f3f86 (at 192.168.202.22@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5602.275385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5602.289310] Lustre: Skipped 1 previous similar message [ 5602.294580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5602.301647] Lustre: Skipped 1 previous similar message [ 5607.394837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5607.402285] Lustre: Skipped 1 previous similar message [ 5607.560083] Lustre: server umount lustre-OST0000 complete [ 5611.230122] LustreError: 168696:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5611.236111] Lustre: 168046:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5611.245601] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5611.256912] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5611.265209] LustreError: 168046:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000081b9ef1a x1873755483716736/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5611.287115] LustreError: 168046:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5611.307352] LustreError: 168046:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5611.318619] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 5611.561547] Lustre: server umount lustre-MDT0000 complete [ 5612.516133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.528075] Lustre: Skipped 1 previous similar message [ 5612.535534] LustreError: 164954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5612.566258] LustreError: 164954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5615.081615] LustreError: 164938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786952622 with bad export cookie 8088691104191680132 [ 5615.084941] LustreError: 164938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5615.092055] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5615.098253] LustreError: 164267:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5617.576569] Lustre: server umount lustre-MDT0001 complete [ 5627.825681] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5631.028457] Key type lgssc unregistered [ 5631.342178] LNet: 169452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5631.355153] LNetError: 169452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5631.368040] LNet: Removed LNI 192.168.202.122@tcp [ 5632.207179] Key type .llcrypt unregistered [ 5632.209464] Key type ._llcrypt unregistered [ 5657.075436] Key type ._llcrypt registered [ 5657.077920] Key type .llcrypt registered [ 5657.267683] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5658.481088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5658.694622] alg: No test for adler32 (adler32-zlib) [ 5659.859714] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 5660.125911] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 5661.815188] Key type lgssc registered [ 5663.164249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5671.520978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5671.556936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.237460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5678.148680] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5683.686133] LustreError: 170865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5683.695629] LustreError: 170865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5687.892709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.349720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5693.951239] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5702.292519] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5707.925020] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5713.812577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5714.390669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5718.511813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5720.308280] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5727.943156] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5738.980575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5738.982715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5738.989811] LustreError: Skipped 1 previous similar message [ 5739.002858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5739.015807] Lustre: Skipped 1 previous similar message [ 5739.030198] Lustre: Skipped 1 previous similar message [ 5740.836566] Lustre: server umount lustre-OST0000 complete [ 5744.612672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5744.635112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5744.656721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5749.223112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5749.225061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5750.923801] Lustre: server umount lustre-MDT0000 complete [ 5754.340146] LustreError: 170860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5754.355484] LustreError: 170860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5755.449492] LustreError: 173646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786952763 with bad export cookie 18413875331217280019 [ 5755.451428] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5755.495266] LustreError: 173646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5755.798072] Lustre: server umount lustre-MDT0001 complete [ 5766.681497] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5770.013726] Key type lgssc unregistered [ 5770.390994] LNet: 174399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5770.407730] LNetError: 174399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5771.432454] LNet: Removed LNI 192.168.202.122@tcp [ 5772.440336] Key type .llcrypt unregistered [ 5772.442819] Key type ._llcrypt unregistered [ 5805.591884] Key type ._llcrypt registered [ 5805.593881] Key type .llcrypt registered [ 5805.687510] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5806.741593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5806.898234] alg: No test for adler32 (adler32-zlib) [ 5807.960970] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 5808.166341] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 5809.815264] Key type lgssc registered [ 5811.016028] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5818.390845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5818.414329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5819.795699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5824.501829] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5830.118078] LustreError: 176104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5830.137275] LustreError: 176104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5833.055851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.516620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5839.301541] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5848.788084] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5856.059695] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5864.996655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5865.476249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5873.976398] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 5882.578338] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5890.047273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5899.233149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5899.242403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5899.260517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5900.267039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5900.268241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5900.278898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5900.300276] Lustre: Skipped 1 previous similar message [ 5905.290509] Lustre: server umount lustre-OST0000 complete [ 5910.497320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5910.499745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5910.514335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5915.618114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5915.624620] Lustre: Skipped 2 previous similar messages [ 5920.737862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5920.751404] Lustre: Skipped 1 previous similar message [ 5923.296667] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5923.470194] Lustre: server umount lustre-MDT0000 complete [ 5925.856932] LustreError: 177889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5925.879735] LustreError: 177889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5927.980057] LustreError: 176085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786952935 with bad export cookie 10227974019429449375 [ 5927.990917] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5927.992687] LustreError: 176085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5928.443847] Lustre: server umount lustre-MDT0001 complete [ 5941.711680] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5944.745669] Key type lgssc unregistered [ 5945.131760] LNet: 179666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5945.136296] LNetError: 179666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5945.162102] LNet: Removed LNI 192.168.202.122@tcp [ 5946.170069] Key type .llcrypt unregistered [ 5946.175045] Key type ._llcrypt unregistered [ 5969.690194] Key type ._llcrypt registered [ 5969.691376] Key type .llcrypt registered [ 5969.853722] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5970.941272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5970.968676] alg: No test for adler32 (adler32-zlib) [ 5971.949848] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 5972.178095] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 5973.863281] Key type lgssc registered [ 5975.292258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5987.195442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5987.203621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5988.674063] Lustre: *** cfs_fail_loc=135, val=0*** [ 5988.819845] LustreError: 181188:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5988.830108] LustreError: 181188:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.122@tcp: cfg command failed: rc = -2 [ 5988.842467] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5988.842467] [ 5988.864572] LustreError: MGC192.168.202.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5988.878383] LustreError: 181149:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5988.893868] LustreError: 181149:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5988.911033] LustreError: 181149:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5988.919834] LustreError: 181149:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5989.066233] Lustre: server umount lustre-MDT0000 complete [ 5989.070198] LustreError: 181149:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5996.783315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5997.516302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6003.040522] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6007.779145] LustreError: 181559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6007.799583] LustreError: 181559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6012.090881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6012.508727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6017.308887] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6025.784489] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6032.493631] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6034.404131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6034.417119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6037.479223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6037.487543] Lustre: Skipped 2 previous similar messages [ 6040.684587] Lustre: server umount lustre-MDT0000 complete [ 6042.593325] LustreError: 181559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6042.619089] LustreError: 181559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6044.910497] LustreError: 181541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953052 with bad export cookie 13910688191160643586 [ 6044.921835] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6044.931924] LustreError: 181541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6045.416370] Lustre: server umount lustre-MDT0001 complete [ 6057.598274] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 6060.604678] Key type lgssc unregistered [ 6061.016625] LNet: 183884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6061.025297] LNetError: 183884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6061.036285] LNet: Removed LNI 192.168.202.122@tcp [ 6062.075655] Key type .llcrypt unregistered [ 6062.078406] Key type ._llcrypt unregistered [ 6085.544943] Key type ._llcrypt registered [ 6085.552568] Key type .llcrypt registered [ 6085.683672] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 6085.987614] Key type .llcrypt unregistered [ 6085.990315] Key type ._llcrypt unregistered [ 6107.936425] Key type ._llcrypt registered [ 6107.938369] Key type .llcrypt registered [ 6108.015139] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 6109.393909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6109.485124] alg: No test for adler32 (adler32-zlib) [ 6110.462988] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 6110.697617] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 6112.407593] Key type lgssc registered [ 6113.989971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6122.659589] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6122.691456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6138.271408] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6143.391531] LustreError: 186021:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.122@tcp: failed processing log, type 4: rc = -110 [ 6170.144591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6176.725440] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6184.143447] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6191.082382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6191.588030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6191.657478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6195.734354] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6196.704616] LustreError: 186046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6196.752193] LustreError: 186046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6201.826721] LustreError: 186047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6203.915782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6204.244589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6208.383315] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6215.384827] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6220.825269] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6224.280483] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6236.133680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6236.155648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6241.249146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6241.255464] Lustre: Skipped 2 previous similar messages [ 6241.892476] Lustre: server umount lustre-OST0000 complete [ 6246.372475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.380120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6246.391025] Lustre: Skipped 2 previous similar messages [ 6251.489880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6251.504952] Lustre: Skipped 2 previous similar messages [ 6251.899303] Lustre: server umount lustre-MDT0000 complete [ 6255.801970] LustreError: 186028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953263 with bad export cookie 8992413030762709412 [ 6255.814383] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6256.003585] Lustre: server umount lustre-MDT0001 complete [ 6270.622742] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 6273.253989] Key type lgssc unregistered [ 6273.548846] LNet: 189990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6273.553970] LNetError: 189990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6273.567375] LNet: Removed LNI 192.168.202.122@tcp [ 6274.393466] Key type .llcrypt unregistered [ 6274.395719] Key type ._llcrypt unregistered [ 6296.393724] Key type ._llcrypt registered [ 6296.397448] Key type .llcrypt registered [ 6296.539082] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 6297.163514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6297.186955] alg: No test for adler32 (adler32-zlib) [ 6298.206844] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 6298.452258] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 6300.087231] Key type lgssc registered [ 6301.251764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6307.511532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6307.539357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6308.965983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6312.915639] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6318.567485] LustreError: 191383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6318.602988] LustreError: 191383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6322.516752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.905854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6327.086429] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6334.047537] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6339.670638] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6346.505613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6346.898445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6353.730395] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6354.169287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6361.647583] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6366.372936] Lustre: Failing over lustre-MDT0000 [ 6366.727135] Lustre: server umount lustre-MDT0000 complete [ 6369.202322] LustreError: 191380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6369.260782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6369.264213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6369.776799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.282021] LustreError: 193531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6374.304387] LustreError: 193531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6379.404200] LustreError: 193531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6379.421702] LustreError: 193531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6384.522085] LustreError: 193531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6384.555328] LustreError: 193531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6385.009299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6385.122996] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6385.463359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6385.553143] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6389.511275] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6389.635598] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6390.766144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6390.815785] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6390.869311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6393.750694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6401.457818] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6409.712084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6409.722118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6409.732944] Lustre: Skipped 1 previous similar message [ 6409.737835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6411.240829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6413.506203] Lustre: server umount lustre-OST0000 complete [ 6421.472032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6421.476094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6421.482025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6421.496161] Lustre: Skipped 2 previous similar messages [ 6421.504936] Lustre: Skipped 2 previous similar messages [ 6426.596574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6431.714361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6431.733508] Lustre: Skipped 2 previous similar messages [ 6432.223249] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6432.756642] Lustre: server umount lustre-MDT0000 complete [ 6436.834535] LustreError: 191384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6436.851581] LustreError: 191384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6436.945726] LustreError: 191365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953444 with bad export cookie 18429015685449596823 [ 6436.961945] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6436.962804] LustreError: 191365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6437.339183] Lustre: server umount lustre-MDT0001 complete [ 6450.420243] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 6454.685882] Key type lgssc unregistered [ 6454.988836] LNet: 196277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6455.007181] LNetError: 196277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6455.015790] LNet: Removed LNI 192.168.202.122@tcp [ 6456.313913] Key type .llcrypt unregistered [ 6456.317347] Key type ._llcrypt unregistered [ 6484.141381] Key type ._llcrypt registered [ 6484.144347] Key type .llcrypt registered [ 6484.305690] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 6485.422383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6485.519559] alg: No test for adler32 (adler32-zlib) [ 6486.568871] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 6486.869719] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 6488.584569] Key type lgssc registered [ 6489.755948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6496.979402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6497.001103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6498.360204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6502.177225] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6508.519534] LustreError: 197695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6508.538466] LustreError: 197695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6510.607503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6510.929568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6515.648807] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6523.166178] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6528.192287] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6535.492170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6535.653983] LustreError: 199516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6536.029475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6542.783400] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6543.370251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6550.718723] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6553.932413] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6558.788949] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6576.568467] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6589.411963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6589.415910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6589.451182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6594.528403] LustreError: 199910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6594.547334] LustreError: 199910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6594.680915] Lustre: server umount lustre-OST0000 complete [ 6603.231897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6603.237379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6603.252240] Lustre: Skipped 1 previous similar message [ 6603.257931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6603.262108] Lustre: Skipped 1 previous similar message [ 6604.769546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6604.770292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.412307] Lustre: server umount lustre-MDT0000 complete [ 6609.887740] LustreError: 197696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6609.907936] LustreError: 197696:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6610.575399] LustreError: 197677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953618 with bad export cookie 6767615400620804052 [ 6610.581844] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6610.588188] LustreError: 197677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6610.994828] Lustre: server umount lustre-MDT0001 complete [ 6624.958540] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 6629.979273] Key type lgssc unregistered [ 6630.586077] LNet: 201568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6630.615413] LNetError: 201568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6630.655571] LNet: Removed LNI 192.168.202.122@tcp [ 6632.424232] Key type .llcrypt unregistered [ 6632.427903] Key type ._llcrypt unregistered [ 6659.280512] Key type ._llcrypt registered [ 6659.282048] Key type .llcrypt registered [ 6659.438939] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 6660.765289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6660.851385] alg: No test for adler32 (adler32-zlib) [ 6661.940492] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 6662.289616] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 6664.015680] Key type lgssc registered [ 6665.266544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6673.938951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6673.962314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6675.747748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6681.879208] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6685.158252] LustreError: 202846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6685.183340] LustreError: 202846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6690.273170] LustreError: 202847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6692.806120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6693.269407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6698.917623] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6708.806370] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6716.111924] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6724.442485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6725.059022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6731.952373] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6740.429859] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6745.597817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6750.376439] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6759.716365] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6766.894848] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6774.550950] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6786.536567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6786.536793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6786.538795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6786.565260] Lustre: Skipped 1 previous similar message [ 6791.661528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6791.670825] Lustre: Skipped 1 previous similar message [ 6796.778733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6800.351239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6800.673392] Lustre: server umount lustre-OST0000 complete [ 6807.011586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6807.031234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6807.036524] Lustre: Skipped 2 previous similar messages [ 6812.134933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6812.846349] Lustre: server umount lustre-MDT0000 complete [ 6817.252392] LustreError: 204281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6817.292977] LustreError: 204281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6817.400414] LustreError: 202827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953825 with bad export cookie 4231478882152631934 [ 6817.407695] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6817.421137] LustreError: 202827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6817.842917] Lustre: server umount lustre-MDT0001 complete [ 6829.894311] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 6831.856446] Key type lgssc unregistered [ 6832.096647] LNet: 207727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6832.103637] LNetError: 207727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6832.113977] LNet: Removed LNI 192.168.202.122@tcp [ 6832.978972] Key type .llcrypt unregistered [ 6832.981261] Key type ._llcrypt unregistered [ 6858.536491] Key type ._llcrypt registered [ 6858.547076] Key type .llcrypt registered [ 6858.718678] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 6861.349541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6861.444151] alg: No test for adler32 (adler32-zlib) [ 6862.681787] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 6863.073256] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 6864.727307] Key type lgssc registered [ 6866.176718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6876.256453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6876.269944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.646924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6883.217470] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6887.905907] LustreError: 209144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6887.943806] LustreError: 209144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6892.665801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6893.007357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6897.934638] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6906.612749] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6913.024694] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6919.861600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6920.465715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6922.553993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6926.998754] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 6935.594549] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6953.443276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6953.461281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6953.468871] Lustre: Skipped 1 previous similar message [ 6954.866313] Lustre: server umount lustre-OST0000 complete [ 6959.587111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6959.603413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.625817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6959.633511] Lustre: Skipped 1 previous similar message [ 6963.682046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6963.682959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6965.701901] Lustre: server umount lustre-MDT0000 complete [ 6968.803532] LustreError: 209141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6968.827680] LustreError: 209141:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6969.203791] LustreError: 209125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953977 with bad export cookie 10078103566787543607 [ 6969.208913] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6969.214732] LustreError: 209125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6969.616497] Lustre: server umount lustre-MDT0001 complete [ 6979.783844] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 6983.002908] Key type lgssc unregistered [ 6983.291405] LNet: 212829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6983.294793] LNetError: 212829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6983.311204] LNet: Removed LNI 192.168.202.122@tcp [ 6983.926260] Key type .llcrypt unregistered [ 6983.930280] Key type ._llcrypt unregistered [ 7006.569849] Key type ._llcrypt registered [ 7006.572232] Key type .llcrypt registered [ 7006.692584] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 7007.848752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7008.061736] alg: No test for adler32 (adler32-zlib) [ 7009.134459] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 7009.345160] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 7010.992102] Key type lgssc registered [ 7012.225721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7020.687678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7020.715981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7022.586342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7026.831499] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 7032.801655] LustreError: 214095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7032.818755] LustreError: 214095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7034.870787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7035.195407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7040.091828] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 7050.495110] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7057.194338] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7065.395900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7066.073201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7068.151638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7074.166578] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 7082.474142] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7098.848757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7098.864218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.881777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7099.874464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7099.881541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7099.884653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.553364] Lustre: server umount lustre-OST0000 complete [ 7107.043347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7107.069127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7107.099655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7107.106454] Lustre: Skipped 1 previous similar message [ 7110.113679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7110.115331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7111.742615] Lustre: server umount lustre-MDT0000 complete [ 7115.236084] LustreError: 214096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7115.264022] LustreError: 214096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7116.286257] LustreError: 216592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786954124 with bad export cookie 7506199845093236657 [ 7116.291923] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7116.300483] LustreError: 216592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7116.659494] Lustre: server umount lustre-MDT0001 complete [ 7130.051484] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 7133.232264] Key type lgssc unregistered [ 7133.506707] LNet: 217784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7133.525930] LNetError: 217784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7133.561588] LNet: Removed LNI 192.168.202.122@tcp [ 7134.869744] Key type .llcrypt unregistered [ 7134.872950] Key type ._llcrypt unregistered [ 7154.396589] Key type ._llcrypt registered [ 7154.401168] Key type .llcrypt registered [ 7154.505463] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 7155.971646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7156.123924] alg: No test for adler32 (adler32-zlib) [ 7157.288988] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 7157.716058] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 7159.431376] Key type lgssc registered [ 7160.755066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7170.165540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7170.207451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7171.876485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7177.846952] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 7181.294956] LustreError: 219055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7181.337582] LustreError: 219055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7186.401304] LustreError: 219056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7186.982941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7187.300104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7192.959777] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 7201.584290] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7208.023436] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7215.750033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7216.313434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7217.336633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7222.867410] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 7231.043369] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7240.823359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7240.947554] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7241.189689] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7249.110059] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 7257.344898] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7259.212299] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7271.465888] Lustre: Permanently deactivating lustre-OST0001 [ 7271.493122] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7278.685693] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7284.485876] Lustre: Permanently reactivating lustre-OST0001 [ 7284.513248] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7284.519292] Lustre: Skipped 2 previous similar messages [ 7286.733609] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7286.747685] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7286.754109] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7286.766022] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7286.816907] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7296.996726] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7296.998141] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7297.024210] LustreError: Skipped 1 previous similar message [ 7297.060119] Lustre: Skipped 1 previous similar message [ 7297.064244] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7297.088276] Lustre: Skipped 1 previous similar message [ 7298.539405] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7298.544712] Lustre: Skipped 1 previous similar message [ 7302.491943] Lustre: server umount lustre-OST0001 complete [ 7303.655393] LustreError: 222861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7303.674138] LustreError: 222861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7308.769988] LustreError: 220876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7308.783438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7308.789711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7308.798273] LustreError: 220876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7308.842612] Lustre: Skipped 2 previous similar messages [ 7312.380326] Lustre: server umount lustre-OST0000 complete [ 7319.014347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7319.022503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7319.055594] Lustre: Skipped 1 previous similar message [ 7319.072540] Lustre: Skipped 1 previous similar message [ 7324.129655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7324.138593] Lustre: Skipped 2 previous similar messages [ 7331.295157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7331.522705] Lustre: server umount lustre-MDT0000 complete [ 7333.346051] LustreError: 219055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7333.363249] LustreError: 219055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7335.460215] LustreError: 219036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786954343 with bad export cookie 16684415050466382878 [ 7335.461539] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7335.466690] LustreError: 219036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7335.880524] Lustre: server umount lustre-MDT0001 complete [ 7348.488677] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 7351.737136] Key type lgssc unregistered [ 7352.146511] LNet: 224108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7352.158521] LNetError: 224108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7352.195699] LNet: Removed LNI 192.168.202.122@tcp [ 7353.328305] Key type .llcrypt unregistered [ 7353.332180] Key type ._llcrypt unregistered [ 7392.812337] Key type ._llcrypt registered [ 7392.818348] Key type .llcrypt registered [ 7392.968206] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 7394.891022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7394.936092] alg: No test for adler32 (adler32-zlib) [ 7396.136715] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 7396.368750] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 7398.023217] Key type lgssc registered [ 7399.509468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7409.419366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7409.448266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7410.707098] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7410.734204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7410.887202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7417.591699] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 7426.680954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7426.806531] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7426.837284] Lustre: 226316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7427.053396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7431.482607] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 7439.092140] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7445.084529] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7452.651565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7452.876078] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7453.134503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.629793] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 7463.054077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7468.646932] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7472.564560] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7479.404988] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7485.010487] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7491.568287] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7509.030873] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7509.036203] Lustre: Skipped 1 previous similar message [ 7530.991597] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7531.003148] Lustre: Skipped 2 previous similar messages [ 7566.699806] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7566.704891] Lustre: Skipped 4 previous similar messages [ 7636.962036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7636.971260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7637.004489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7637.491130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7637.503735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7637.533545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7637.559515] Lustre: Skipped 1 previous similar message [ 7640.598797] Lustre: server umount lustre-OST0000 complete [ 7647.200486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7647.210740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.236254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7651.511995] Lustre: server umount lustre-MDT0000 complete [ 7655.700551] LustreError: 229380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786954663 with bad export cookie 8900888602756259345 [ 7655.705486] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7655.712227] LustreError: 229380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7656.062720] Lustre: server umount lustre-MDT0001 complete [ 7669.334840] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 7672.216349] Key type lgssc unregistered [ 7672.550574] LNet: 231110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7672.558762] LNetError: 231110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7672.588629] LNet: Removed LNI 192.168.202.122@tcp [ 7673.596984] Key type .llcrypt unregistered [ 7673.601382] Key type ._llcrypt unregistered [ 7700.059586] Key type ._llcrypt registered [ 7700.062298] Key type .llcrypt registered [ 7700.201810] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 7701.917781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7701.977634] alg: No test for adler32 (adler32-zlib) [ 7703.089954] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 7703.350771] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 7704.991181] Key type lgssc registered [ 7706.267909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7715.149428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7715.191090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7716.900427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7722.306629] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 7727.095425] LustreError: 232525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7727.106593] LustreError: 232525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7731.589659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7731.926824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7737.303832] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 7745.428799] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7752.145770] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7759.322547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7759.765927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7761.811031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7766.624027] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all [ 7774.853665] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7802.849834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7802.873586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7802.873698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7805.319169] Lustre: server umount lustre-OST0000 complete [ 7813.091290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.098780] Lustre: Skipped 1 previous similar message [ 7813.102728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7813.116779] Lustre: Skipped 2 previous similar messages [ 7818.213429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7818.230418] Lustre: Skipped 1 previous similar message [ 7822.305640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7824.351144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7824.474798] LustreError: 235472:0:(ldlm_resource.c:1208:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (0000000089fb8852) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7824.643961] Lustre: server umount lustre-MDT0000 complete [ 7827.428952] LustreError: 234468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7827.450486] LustreError: 234468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7828.623540] LustreError: 232505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786954836 with bad export cookie 5262682665442978058 [ 7828.628918] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7828.638804] LustreError: 232505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7829.025537] Lustre: server umount lustre-MDT0001 complete [ 7840.715680] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 7844.137576] Key type lgssc unregistered [ 7844.575952] LNet: 236229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7844.584321] LNetError: 236229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7844.598280] LNet: Removed LNI 192.168.202.122@tcp [ 7845.671701] Key type .llcrypt unregistered [ 7845.677344] Key type ._llcrypt unregistered [ 7874.106190] Key type ._llcrypt registered [ 7874.108154] Key type .llcrypt registered [ 7874.296073] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 7875.854159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7875.899617] alg: No test for adler32 (adler32-zlib) [ 7877.024467] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 7877.229311] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 7878.920950] Key type lgssc registered [ 7880.359870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8151.972840] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 8155.297461] Key type lgssc unregistered [ 8155.622815] LNet: 238258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8155.638443] LNetError: 238258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8155.667682] LNet: Removed LNI 192.168.202.122@tcp [ 8156.679421] Key type .llcrypt unregistered [ 8156.682935] Key type ._llcrypt unregistered [ 8186.500196] Key type ._llcrypt registered [ 8186.502452] Key type .llcrypt registered [ 8186.585754] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 8203.206937] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 8205.152736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8205.210192] alg: No test for adler32 (adler32-zlib) [ 8206.379287] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 8206.754955] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 8208.455551] Key type lgssc registered [ 8210.152814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8250.480647] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7885 sec ========== 04:27:36 (1786955256) [ 8252.248116] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:27:38 (1786955258) === [ 8256.169542] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:27:41 (1786955261) === [ 8304.853478] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 8308.054728] Key type lgssc unregistered [ 8308.403147] LNet: 243502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8308.414060] LNetError: 243502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8308.456825] LNet: Removed LNI 192.168.202.122@tcp [ 8309.443602] Key type .llcrypt unregistered [ 8309.445131] Key type ._llcrypt unregistered