[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 511936959 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003376] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.011043] LSM: Security Framework initializing [ 0.012000] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.013000] *** VALIDATE selinux *** [ 0.020058] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025181] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028100] *** VALIDATE tmpfs *** [ 0.029445] *** VALIDATE proc *** [ 0.031203] *** VALIDATE cgroup *** [ 0.032008] *** VALIDATE cgroup2 *** [ 0.033267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036027] Spectre V2 : User space: Vulnerable [ 0.037009] Speculative Store Bypass: Vulnerable [ 0.040507] debug: unmapping init [mem 0xffffffff8dc59000-0xffffffff8dc60fff] [ 0.042150] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043704] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044024] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046017] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048011] ... max period: 00007fffffffffff [ 0.049012] ... fixed-purpose events: 3 [ 0.050010] ... event mask: 000000070000000f [ 0.051275] rcu: Hierarchical SRCU implementation. [ 0.053394] smp: Bringing up secondary CPUs ... [ 0.054578] x86: Booting SMP configuration: [ 0.055028] .... node #0, CPUs: #1 #2 #3 [ 0.062063] smp: Brought up 1 node, 4 CPUs [ 0.064011] smpboot: Max logical packages: 1 [ 0.065021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132916] node 0 deferred pages initialised in 66ms [ 0.137030] devtmpfs: initialized [ 0.138324] x86/mm: Memory block size: 128MB [ 0.140256] gcov: version magic: 0x41383552 [ 0.142390] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146328] pinctrl core: initialized pinctrl subsystem [ 0.148162] [ 0.148517] ************************************************************* [ 0.150011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152009] ** ** [ 0.155012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157012] ** ** [ 0.160011] ** This means that this kernel is built to expose internal ** [ 0.164015] ** IOMMU data structures, which may compromise security on ** [ 0.166011] ** your system. ** [ 0.169014] ** ** [ 0.171007] ** If you see this message and you are not debugging the ** [ 0.173013] ** kernel, report this immediately to your vendor! ** [ 0.175021] ** ** [ 0.178012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180012] ************************************************************* [ 0.182587] NET: Registered protocol family 16 [ 0.185600] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.191086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196111] cpuidle: using governor menu [ 0.197914] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201749] PCI: Using configuration type 1 for base access [ 0.204156] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.213204] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.217104] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.221142] cryptd: max_cpu_qlen set to 1000 [ 0.224382] ACPI: Added _OSI(Module Device) [ 0.226021] ACPI: Added _OSI(Processor Device) [ 0.227015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230026] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235860] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242584] ACPI: Interpreter enabled [ 0.244063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246020] ACPI: Using IOAPIC for interrupt routing [ 0.248157] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.251432] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.262583] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.265130] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.269030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.275109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281796] acpiphp: Slot [2] registered [ 0.284179] acpiphp: Slot [5] registered [ 0.286185] acpiphp: Slot [6] registered [ 0.287237] acpiphp: Slot [7] registered [ 0.289195] acpiphp: Slot [8] registered [ 0.291186] acpiphp: Slot [9] registered [ 0.293220] acpiphp: Slot [10] registered [ 0.294176] acpiphp: Slot [3] registered [ 0.296144] acpiphp: Slot [4] registered [ 0.298110] acpiphp: Slot [11] registered [ 0.300154] acpiphp: Slot [12] registered [ 0.301145] acpiphp: Slot [13] registered [ 0.303186] acpiphp: Slot [14] registered [ 0.305124] acpiphp: Slot [15] registered [ 0.306115] acpiphp: Slot [16] registered [ 0.308112] acpiphp: Slot [17] registered [ 0.309112] acpiphp: Slot [18] registered [ 0.310127] acpiphp: Slot [19] registered [ 0.312118] acpiphp: Slot [20] registered [ 0.313089] acpiphp: Slot [21] registered [ 0.314135] acpiphp: Slot [22] registered [ 0.316108] acpiphp: Slot [23] registered [ 0.317122] acpiphp: Slot [24] registered [ 0.319112] acpiphp: Slot [25] registered [ 0.320142] acpiphp: Slot [26] registered [ 0.322138] acpiphp: Slot [27] registered [ 0.323114] acpiphp: Slot [28] registered [ 0.325132] acpiphp: Slot [29] registered [ 0.326114] acpiphp: Slot [30] registered [ 0.328146] acpiphp: Slot [31] registered [ 0.329090] PCI host bridge to bus 0000:00 [ 0.331024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.333028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.335028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.338029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.340032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.343030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345210] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348064] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351364] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.361686] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.366601] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.371015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.373020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377058] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.379848] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.385763] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.392023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.405018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.411764] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416925] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.424020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.432022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.450019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.460611] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.467027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.474019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.510016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.529820] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.537022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.543021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.560000] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.570934] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.582016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.594018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.619024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.630856] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.637021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.644020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.663020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.671888] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.680015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.686019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.699021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.711556] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.714406] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.716412] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.719443] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.722257] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.726143] iommu: Default domain type: Passthrough [ 0.728446] SCSI subsystem initialized [ 0.730145] ACPI: bus type USB registered [ 0.733154] usbcore: registered new interface driver usbfs [ 0.735091] usbcore: registered new interface driver hub [ 0.737089] usbcore: registered new device driver usb [ 0.739315] pps_core: LinuxPPS API ver. 1 registered [ 0.742020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.746137] PTP clock support registered [ 0.748131] EDAC MC: Ver: 3.0.0 [ 0.750227] PCI: Using ACPI for IRQ routing [ 0.751548] NetLabel: Initializing [ 0.752000] NetLabel: domain hash size = 128 [ 0.755018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.757076] NetLabel: unlabeled traffic allowed by default [ 0.759145] vgaarb: loaded [ 0.761213] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.763016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.770327] clocksource: Switched to clocksource kvm-clock [ 0.884054] VFS: Disk quotas dquot_6.6.0 [ 0.885706] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.888597] *** VALIDATE ramfs *** [ 0.890139] *** VALIDATE hugetlbfs *** [ 0.892139] pnp: PnP ACPI init [ 0.895241] pnp: PnP ACPI: found 6 devices [ 0.914746] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.918485] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.921187] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.923789] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.926601] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.929505] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.932953] NET: Registered protocol family 2 [ 0.935868] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.941351] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.945565] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.951613] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.955952] TCP: Hash tables configured (established 65536 bind 65536) [ 0.960419] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.963675] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.966213] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.969533] NET: Registered protocol family 1 [ 0.972383] RPC: Registered named UNIX socket transport module. [ 0.975422] RPC: Registered udp transport module. [ 0.977551] RPC: Registered tcp transport module. [ 0.979719] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.982871] NET: Registered protocol family 44 [ 0.984896] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.987496] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.989898] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.992619] PCI: CLS 0 bytes, default 64 [ 0.995221] Unpacking initramfs... [ 2.411898] debug: unmapping init [mem 0xffff95f2bcc54000-0xffff95f2bffbffff] [ 2.418493] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.421037] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.425500] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.918411] Initialise system trusted keyrings [ 2.920269] Key type blacklist registered [ 2.922621] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.934468] zbud: loaded [ 2.938137] *** VALIDATE nfs *** [ 2.939660] *** VALIDATE nfs4 *** [ 2.941195] pstore: using deflate compression [ 2.944739] Platform Keyring initialized [ 3.045515] NET: Registered protocol family 38 [ 3.047557] Key type asymmetric registered [ 3.049359] Asymmetric key parser 'x509' registered [ 3.051437] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.054593] io scheduler mq-deadline registered [ 3.056720] io scheduler kyber registered [ 3.058688] io scheduler bfq registered [ 3.060679] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.064474] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.067637] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.071065] ACPI: Power Button [PWRF] [ 3.076440] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.084816] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.096215] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.102955] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.118022] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.146304] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.176181] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.180800] Non-volatile memory driver v1.3 [ 3.182578] Linux agpgart interface v0.103 [ 3.215392] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.219022] vda: detected capacity change from 0 to 74702848 [ 3.245903] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.250645] vdb: detected capacity change from 0 to 1073741824 [ 3.275713] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.279975] vdc: detected capacity change from 0 to 2621440000 [ 3.296624] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.299819] vdd: detected capacity change from 0 to 2621440000 [ 3.319089] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.322756] vde: detected capacity change from 0 to 4294967296 [ 3.337385] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.340801] vdf: detected capacity change from 0 to 4294967296 [ 3.347860] libphy: Fixed MDIO Bus: probed [ 3.354485] usbcore: registered new interface driver usbserial_generic [ 3.357412] usbserial: USB Serial support registered for generic [ 3.359847] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.364418] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.366625] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.369360] mousedev: PS/2 mouse device common for all mice [ 3.371907] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.374674] rtc_cmos 00:05: RTC can wake from S4 [ 3.378383] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.378937] rtc_cmos 00:05: registered as rtc0 [ 3.383555] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.384231] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.387748] intel_pstate: CPU model not supported [ 3.392549] hid: raw HID events driver (C) Jiri Kosina [ 3.394972] usbcore: registered new interface driver usbhid [ 3.396907] usbhid: USB HID core driver [ 3.398410] drop_monitor: Initializing network drop monitor service [ 3.400721] Initializing XFRM netlink socket [ 3.402808] NET: Registered protocol family 10 [ 3.406212] Segment Routing with IPv6 [ 3.407301] NET: Registered protocol family 17 [ 3.409053] mpls_gso: MPLS GSO support [ 3.415264] RAS: Correctable Errors collector initialized. [ 3.417516] AVX version of gcm_enc/dec engaged. [ 3.419262] AES CTR mode by8 optimization enabled [ 3.500581] sched_clock: Marking stable (3500552737, 0)->(4449360348, -948807611) [ 3.504539] registered taskstats version 1 [ 3.507383] Loading compiled-in X.509 certificates [ 3.510517] zswap: loaded using pool lzo/zbud [ 3.540108] Key type big_key registered [ 3.553676] Key type encrypted registered [ 3.555595] ima: No TPM chip found, activating TPM-bypass! [ 3.557379] ima: Allocated hash algorithm: sha1 [ 3.559514] ima: No architecture policies found [ 3.561555] evm: Initialising EVM extended attributes: [ 3.563618] evm: security.selinux [ 3.565020] evm: security.ima [ 3.566478] evm: security.capability [ 3.567893] evm: HMAC attrs: 0x1 [ 3.570726] rtc_cmos 00:05: setting system clock to 2026-08-15 20:27:28 UTC (1786825648) [ 3.576690] debug: unmapping init [mem 0xffffffff8ec03000-0xffffffff8edfffff] [ 3.579890] debug: unmapping init [mem 0xffffffff8d982000-0xffffffff8dc58fff] [ 3.588127] Write protecting the kernel read-only data: 28672k [ 3.591433] debug: unmapping init [mem 0xffffffff8c003000-0xffffffff8c1fffff] [ 3.594063] debug: unmapping init [mem 0xffffffff8c914000-0xffffffff8c9fffff] [ 3.630456] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.638815] systemd[1]: Detected virtualization kvm. [ 3.640835] systemd[1]: Detected architecture x86-64. [ 3.642653] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.670708] systemd[1]: No hostname configured. [ 3.672759] systemd[1]: Set hostname to . [ 3.675238] random: systemd: uninitialized urandom read (16 bytes read) [ 3.678181] systemd[1]: Initializing machine ID from random generator. [ 3.711452] random: ln: uninitialized urandom read (6 bytes read) [ 3.823098] random: systemd: uninitialized urandom read (16 bytes read) [ 3.826426] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.830608] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.838988] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.450975] device-mapper: uevent: version 1.0.3 [ 4.453732] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook.[ 4.840672] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.241777] virtio_net virtio0 ens2: renamed from eth0 [ 5.360232] scsi host0: ata_piix [ 5.378558] scsi host1: ata_piix [ 5.380432] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.383348] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.066612] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.074259] random: crng init done [ 10.077194] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.498536] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.598132] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.858759] SELinux: Disabled at runtime. [ 11.917535] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.927253] systemd[1]: Detected virtualization kvm. [ 11.929135] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.370434] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.373877] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.379040] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.382667] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.386224] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.393350] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.417693] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 12.493228] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.841744] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.100324] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.112452] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.197799] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.223646] EDAC sbridge: Ver: 1.1.2 [ 14.949191] Key type dns_resolver registered [ 15.257367] NFS: Registering the id_resolver key type [ 15.259631] Key type id_resolver registered [ 15.261729] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg305-server login: [ 39.636360] libcfs: loading out-of-tree module taints kernel. [ 39.660360] Key type ._llcrypt registered [ 39.662279] Key type .llcrypt registered [ 39.716160] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 52.007738] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 53.466031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.484211] alg: No test for adler32 (adler32-zlib) [ 54.901543] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 55.676613] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 57.399360] Key type lgssc registered [ 58.786174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.306749] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 108.826795] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 110.235006] hrtimer: interrupt took 3919015 ns [ 120.786289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 120.815876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 122.198886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 122.248924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 122.406560] Lustre: lustre-MDT0000: new disk, initializing [ 122.512277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 122.538585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 126.993870] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 141.590991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 141.805629] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 141.877549] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 141.886356] Lustre: Skipped 1 previous similar message [ 142.110052] Lustre: lustre-MDT0001: new disk, initializing [ 142.245285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 142.276743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 142.306664] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 146.891926] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 151.796095] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 161.606180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 161.830431] Lustre: lustre-OST0000: new disk, initializing [ 161.834658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 161.841030] Lustre: 8423:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 161.933930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.290358] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.504677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 169.513923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 169.585417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 179.634223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 179.751687] Lustre: lustre-OST0001: new disk, initializing [ 179.755504] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 179.759945] Lustre: 9495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 179.822330] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 186.531945] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.010898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 190.027146] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 190.112276] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 197.810996] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 202.286472] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 207.300420] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing check_logdir /tmp/testlogs/ [ 211.226239] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing yml_node [ 214.840485] Lustre: DEBUG MARKER: Client: 2.17.57.5 [ 216.957871] Lustre: DEBUG MARKER: MDS: 2.17.57.5 [ 218.990737] Lustre: DEBUG MARKER: OSS: 2.17.57.5 [ 220.277866] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 16:31:03 EDT 2026 [ 234.245514] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 235.461594] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 251.872096] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 251.873263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 251.893467] Lustre: Skipped 3 previous similar messages [ 251.902545] Lustre: Skipped 2 previous similar messages [ 256.992205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 256.994497] Lustre: Skipped 3 previous similar messages [ 262.113940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 262.121919] Lustre: Skipped 3 previous similar messages [ 264.159336] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 264.456117] Lustre: server umount lustre-MDT0000 complete [ 267.575810] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825912 with bad export cookie 11921305084414397198 [ 267.577042] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 267.585603] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 272.351864] LustreError: 12236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 272.352757] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 272.353941] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 272.366071] LustreError: 12236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 272.401159] Lustre: Skipped 1 previous similar message [ 273.908708] Lustre: server umount lustre-MDT0001 complete [ 277.513171] Lustre: server umount lustre-OST0000 complete [ 281.018317] Lustre: server umount lustre-OST0001 complete [ 287.676962] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 295.441725] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 334.403611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 334.743527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 334.796664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 334.911635] Lustre: lustre-MDT0000: new disk, initializing [ 335.086761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 335.129446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 339.670146] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 349.103751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 349.225283] Lustre: 15044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 349.306779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 349.311160] Lustre: Skipped 1 previous similar message [ 349.385284] Lustre: lustre-MDT0001: new disk, initializing [ 349.471747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 349.501849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 349.512726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 353.002631] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 361.275237] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 366.390791] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 372.302971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 372.587742] Lustre: lustre-OST0000: new disk, initializing [ 372.593288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 372.600736] Lustre: 17016:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 372.706212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 373.810866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 373.828473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 373.949596] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 378.786481] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 387.560771] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 391.811829] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 392.133143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 396.287870] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 396.452978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 399.848913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.854956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 399.859655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 399.859676] Lustre: Skipped 1 previous similar message [ 399.876167] Lustre: Skipped 1 previous similar message [ 399.884061] LustreError: Skipped 1 previous similar message [ 410.089087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 410.101693] Lustre: Skipped 4 previous similar messages [ 413.151159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 413.316316] Lustre: server umount lustre-OST0000 complete [ 420.329510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.347852] Lustre: Skipped 1 previous similar message [ 422.927072] Lustre: server umount lustre-MDT0000 complete [ 425.447550] LustreError: 15051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 425.463117] LustreError: 15051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 426.045857] LustreError: 15037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826070 with bad export cookie 11921305084414398731 [ 426.063950] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 426.324712] Lustre: server umount lustre-MDT0001 complete [ 429.150868] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 16:34:32 (1786826072) [ 434.747034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.021104] LustreError: 19281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.066466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.263558] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 440.293346] LustreError: 19282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.388289] LustreError: 19281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 447.535260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.887933] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 459.671351] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 465.211997] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 471.016361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 476.626850] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 483.704358] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 489.967198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 489.969968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.993666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 490.001259] Lustre: Skipped 1 previous similar message [ 490.015616] Lustre: Skipped 3 previous similar messages [ 495.755545] Lustre: server umount lustre-OST0000 complete [ 499.169208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 499.181882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.855660] Lustre: server umount lustre-MDT0000 complete [ 505.314215] LustreError: 19278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.346781] LustreError: 19278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 508.285342] LustreError: 19261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826153 with bad export cookie 11921305084414399613 [ 508.308513] LustreError: 19261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 508.325242] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 508.640675] Lustre: server umount lustre-MDT0001 complete [ 517.494804] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 519.943065] Key type lgssc unregistered [ 520.230876] LNet: 22814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 520.240391] LNetError: 22814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 520.254774] LNet: Removed LNI 192.168.203.105@tcp [ 521.054131] Key type .llcrypt unregistered [ 521.056787] Key type ._llcrypt unregistered [ 542.905849] Key type ._llcrypt registered [ 542.907804] Key type .llcrypt registered [ 542.999273] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 544.068995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 544.280563] alg: No test for adler32 (adler32-zlib) [ 545.346293] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 545.560963] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 547.335449] Key type lgssc registered [ 548.369942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 554.621619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 554.634744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.939871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 559.546664] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 565.226656] LustreError: 24225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.247696] LustreError: 24225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 566.579645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.823745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 570.046576] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 577.077376] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 582.613648] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 588.863238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 589.300448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.720026] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 600.649238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 601.055903] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 615.911890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.923052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 615.933036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 617.499269] Lustre: server umount lustre-OST0000 complete [ 621.025127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.039764] Lustre: Skipped 1 previous similar message [ 621.050143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 621.056287] Lustre: Skipped 1 previous similar message [ 626.145931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 626.150674] Lustre: Skipped 1 previous similar message [ 631.264963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 631.279780] Lustre: Skipped 1 previous similar message [ 635.359134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 635.368094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 635.387898] Lustre: Skipped 1 previous similar message [ 635.640145] Lustre: server umount lustre-MDT0000 complete [ 639.001378] LustreError: 24206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826283 with bad export cookie 8831918171157255827 [ 639.007812] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.015932] LustreError: 24206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 639.293682] Lustre: server umount lustre-MDT0001 complete [ 652.059346] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 654.986055] Key type lgssc unregistered [ 655.196968] LNet: 28203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.203620] LNetError: 28203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.219831] LNet: Removed LNI 192.168.203.105@tcp [ 655.973679] Key type .llcrypt unregistered [ 655.976297] Key type ._llcrypt unregistered [ 679.290769] Key type ._llcrypt registered [ 679.294914] Key type .llcrypt registered [ 679.403164] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 680.454175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 680.472302] alg: No test for adler32 (adler32-zlib) [ 681.452928] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 681.683317] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 683.375192] Key type lgssc registered [ 684.331244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 691.249410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 691.289571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 692.593084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.353997] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 702.952253] LustreError: 29619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.982408] LustreError: 29619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 703.961639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.201325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 707.990435] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 714.717138] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 720.092357] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 732.444097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 732.772689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 737.766322] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 737.904209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 744.302635] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 755.684815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.686375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 755.709773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 755.730387] Lustre: Skipped 1 previous similar message [ 756.780769] Lustre: server umount lustre-OST0000 complete [ 760.801647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.822846] Lustre: Skipped 1 previous similar message [ 760.844848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 760.854779] Lustre: Skipped 1 previous similar message [ 765.923105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 765.928043] Lustre: Skipped 1 previous similar message [ 771.043583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 771.046430] Lustre: Skipped 1 previous similar message [ 775.135103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 775.143914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 775.152415] Lustre: Skipped 1 previous similar message [ 775.309540] Lustre: server umount lustre-MDT0000 complete [ 778.189212] LustreError: 31806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826423 with bad export cookie 4675944068509812623 [ 778.192250] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 778.196544] LustreError: 31806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 778.424974] Lustre: server umount lustre-MDT0001 complete [ 788.015236] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 790.269726] Key type lgssc unregistered [ 790.447921] LNet: 33498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.451709] LNetError: 33498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.463642] LNet: Removed LNI 192.168.203.105@tcp [ 791.135349] Key type .llcrypt unregistered [ 791.137421] Key type ._llcrypt unregistered [ 809.898535] Key type ._llcrypt registered [ 809.901356] Key type .llcrypt registered [ 809.992970] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 810.704698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 810.793950] alg: No test for adler32 (adler32-zlib) [ 811.866917] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 812.061305] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 813.735193] Key type lgssc registered [ 814.641321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 821.970299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 821.990942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.335605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 827.360186] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 833.505544] LustreError: 34889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 833.522783] LustreError: 34889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 835.138405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 835.404820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 839.797145] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 847.931474] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 854.366979] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 861.123097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 861.481531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 868.025583] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 868.663566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 874.938850] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 886.246579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 886.247609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 886.258299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 886.279236] Lustre: Skipped 1 previous similar message [ 888.137954] Lustre: server umount lustre-OST0000 complete [ 896.481063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 896.484257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 896.491833] Lustre: Skipped 1 previous similar message [ 896.499832] Lustre: Skipped 1 previous similar message [ 898.513708] Lustre: server umount lustre-MDT0000 complete [ 901.604297] LustreError: 34889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 901.623809] LustreError: 34889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 901.632135] LustreError: 34870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826546 with bad export cookie 15319881429888999426 [ 901.645828] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 901.884855] Lustre: server umount lustre-MDT0001 complete [ 911.913318] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 914.739279] Key type lgssc unregistered [ 914.994667] LNet: 38426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.002272] LNetError: 38426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.026768] LNet: Removed LNI 192.168.203.105@tcp [ 915.785948] Key type .llcrypt unregistered [ 915.790251] Key type ._llcrypt unregistered [ 937.236573] Key type ._llcrypt registered [ 937.238822] Key type .llcrypt registered [ 937.325252] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 938.293122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 938.377218] alg: No test for adler32 (adler32-zlib) [ 939.369484] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 939.548846] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 941.191180] Key type lgssc registered [ 942.211512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 948.597768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 948.612346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 950.008563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.107878] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 959.460969] LustreError: 39840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.478799] LustreError: 39840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 962.004339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.414637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 966.389420] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 972.846806] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 977.781852] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 984.024576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 984.240639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 989.103561] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 994.485712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 995.244984] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1002.893671] Lustre: lustre-OST0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 1002.981879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1002.991369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.030124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1005.030278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.030826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1005.038406] Lustre: Skipped 1 previous similar message [ 1007.963408] Lustre: lustre-OST0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 1007.973627] Lustre: Skipped 1 previous similar message [ 1010.163479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1010.175051] Lustre: Skipped 1 previous similar message [ 1015.271450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1015.279716] Lustre: Skipped 2 previous similar messages [ 1016.287226] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1016.467581] Lustre: server umount lustre-OST0000 complete [ 1034.209556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1034.219985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1034.240720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1040.167321] Lustre: server umount lustre-MDT0000 complete [ 1040.864260] LustreError: 40567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.886312] LustreError: 40567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1043.527763] LustreError: 39820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826688 with bad export cookie 7083484554231007518 [ 1043.530311] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1043.536664] LustreError: 39820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1050.015528] Lustre: server umount lustre-MDT0001 complete [ 1060.689208] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1063.252298] Key type lgssc unregistered [ 1063.519038] LNet: 43558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.530126] LNetError: 43558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.547915] LNet: Removed LNI 192.168.203.105@tcp [ 1064.346587] Key type .llcrypt unregistered [ 1064.348884] Key type ._llcrypt unregistered [ 1084.774876] Key type ._llcrypt registered [ 1084.776815] Key type .llcrypt registered [ 1084.872323] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1085.905263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1085.928541] alg: No test for adler32 (adler32-zlib) [ 1086.926797] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 1087.150237] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1088.847189] Key type lgssc registered [ 1090.101532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1097.077296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1097.094847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.578940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1103.083155] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1108.966606] LustreError: 44949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1108.993142] LustreError: 44949:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1111.443113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.805512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1115.386649] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1122.241771] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1126.864768] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1133.137465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1133.548790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1138.438977] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1143.794760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1144.864618] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1152.370808] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 1152.993078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1152.999571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.013486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1154.016676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1154.021063] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.042929] Lustre: Skipped 1 previous similar message [ 1157.465636] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 1157.472397] Lustre: Skipped 2 previous similar messages [ 1158.857990] Lustre: server umount lustre-MDT0000 complete [ 1159.168137] LustreError: 44949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1159.198526] LustreError: 44949:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1162.583881] LustreError: 44946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.940136] LustreError: 44929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826807 with bad export cookie 16948695174667009524 [ 1162.958529] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1164.257271] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.266142] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1167.269536] LustreError: 46788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.295030] LustreError: 46788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1169.389625] Lustre: server umount lustre-MDT0001 complete [ 1189.931378] Lustre: server umount lustre-OST0000 complete [ 1204.603485] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1206.772310] Key type lgssc unregistered [ 1207.047752] LNet: 48813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.052873] LNetError: 48813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.078310] LNet: Removed LNI 192.168.203.105@tcp [ 1208.011520] Key type .llcrypt unregistered [ 1208.014889] Key type ._llcrypt unregistered [ 1228.594565] Key type ._llcrypt registered [ 1228.597328] Key type .llcrypt registered [ 1228.695211] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1229.467832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1229.633800] alg: No test for adler32 (adler32-zlib) [ 1230.646922] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 1230.930541] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1232.585539] Key type lgssc registered [ 1233.674396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1240.441791] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1240.469253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1255.967472] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1261.087622] LustreError: 50178:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 1287.711228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.849392] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1299.035437] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1466.267759] Lustre: server umount lustre-OST0000 complete [ 1478.643127] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1480.861875] Key type lgssc unregistered [ 1481.052369] LNet: 51856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.056047] LNetError: 51856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.071774] LNet: Removed LNI 192.168.203.105@tcp [ 1481.676963] Key type .llcrypt unregistered [ 1481.683242] Key type ._llcrypt unregistered [ 1501.076565] Key type ._llcrypt registered [ 1501.079300] Key type .llcrypt registered [ 1501.221918] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1502.034173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1502.073853] alg: No test for adler32 (adler32-zlib) [ 1503.178622] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 1503.377394] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1505.015445] Key type lgssc registered [ 1506.115178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1512.881885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1512.902106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1514.365261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.726413] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1524.712456] LustreError: 53248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1524.734059] LustreError: 53248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1524.912442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1525.258347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1528.755847] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1536.076400] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1542.541542] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1549.662052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1550.177511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1556.933082] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1559.410807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1563.623571] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1566.689125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1566.699923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.724359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1569.767384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1569.768881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1569.771864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1569.809740] Lustre: Skipped 1 previous similar message [ 1572.305569] Lustre: server umount lustre-OST0000 complete [ 1576.416951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1576.422427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.433763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1580.000770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1580.008122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1585.120707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1585.136889] Lustre: Skipped 2 previous similar messages [ 1590.239158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1590.355326] Lustre: server umount lustre-MDT0000 complete [ 1593.105392] LustreError: 53227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827238 with bad export cookie 10327736931085050533 [ 1593.125568] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1593.127439] LustreError: 53227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1593.489388] Lustre: server umount lustre-MDT0001 complete [ 1604.366526] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1607.097489] Key type lgssc unregistered [ 1607.358824] LNet: 56782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1607.365267] LNetError: 56782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.377803] LNet: Removed LNI 192.168.203.105@tcp [ 1608.166973] Key type .llcrypt unregistered [ 1608.169096] Key type ._llcrypt unregistered [ 1629.234658] Key type ._llcrypt registered [ 1629.236913] Key type .llcrypt registered [ 1629.339720] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1630.402719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1630.434693] alg: No test for adler32 (adler32-zlib) [ 1631.488218] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 1631.781736] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1633.471300] Key type lgssc registered [ 1634.503397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1641.499936] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1641.520775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1657.055351] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1662.179108] LustreError: 58149:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 1688.799697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.628390] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1702.559368] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1708.431628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1708.975910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.053311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1712.698527] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1714.149567] LustreError: 58175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.178090] LustreError: 58175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1719.285450] LustreError: 58174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1721.100961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.646655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1725.055511] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1735.421984] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1742.726399] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1744.869380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1744.875432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.887826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1747.428246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1747.434633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1747.459866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1749.479354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1754.623326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1754.639381] Lustre: Skipped 2 previous similar messages [ 1759.199362] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1759.442850] Lustre: server umount lustre-OST0000 complete [ 1780.041448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.050346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1780.061747] Lustre: Skipped 1 previous similar message [ 1780.078405] Lustre: Skipped 1 previous similar message [ 1786.457787] Lustre: server umount lustre-MDT0000 complete [ 1789.411419] LustreError: 59186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1789.461147] LustreError: 59186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1790.387076] LustreError: 58155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827435 with bad export cookie 10947655329128505493 [ 1790.391146] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1790.397135] LustreError: 58155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1796.837429] Lustre: server umount lustre-MDT0001 complete [ 1806.096250] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1809.087623] Key type lgssc unregistered [ 1809.391096] LNet: 61899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.402398] LNetError: 61899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1809.413751] LNet: Removed LNI 192.168.203.105@tcp [ 1810.292314] Key type .llcrypt unregistered [ 1810.294433] Key type ._llcrypt unregistered [ 1838.678464] Key type ._llcrypt registered [ 1838.681806] Key type .llcrypt registered [ 1838.783219] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1839.697587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1839.781123] alg: No test for adler32 (adler32-zlib) [ 1840.855765] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 1841.040585] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1842.783326] Key type lgssc registered [ 1843.778553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1850.850225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1850.894982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1852.341936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.875116] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1862.627789] LustreError: 63310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1862.650496] LustreError: 63310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1863.421865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.701804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1867.321472] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1874.037992] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1879.361736] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1886.209226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1886.513947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.915489] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1897.782259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1899.708154] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1927.402608] Lustre: server umount lustre-OST0000 complete [ 1933.810954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1933.819201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1933.837068] Lustre: Skipped 1 previous similar message [ 1933.853280] Lustre: Skipped 1 previous similar message [ 1938.912696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1938.923150] Lustre: Skipped 1 previous similar message [ 1941.108347] Lustre: server umount lustre-MDT0000 complete [ 1944.037314] LustreError: 63310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1944.056489] LustreError: 63310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1944.740255] LustreError: 63291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827589 with bad export cookie 4225817815212578899 [ 1944.757083] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1954.387839] Lustre: server umount lustre-MDT0001 complete [ 1962.271251] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1964.543877] Key type lgssc unregistered [ 1964.776578] LNet: 66869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.782917] LNetError: 66869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.813609] LNet: Removed LNI 192.168.203.105@tcp [ 1965.573272] Key type .llcrypt unregistered [ 1965.575752] Key type ._llcrypt unregistered [ 1987.712429] Key type ._llcrypt registered [ 1987.714584] Key type .llcrypt registered [ 1987.783954] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1988.407778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1988.425258] alg: No test for adler32 (adler32-zlib) [ 1989.506685] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 1989.722765] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1991.375211] Key type lgssc registered [ 1992.319412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1999.947975] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 17:00:43 (1786827643) [ 2005.674165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2005.709756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.207470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.590978] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2017.625857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.770475] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2017.892352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2021.611697] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2029.588318] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2035.613323] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2042.616595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2042.861511] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2042.955499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.835670] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2053.105836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2057.735489] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2063.153813] Lustre: Failing over lustre-MDT0000 [ 2063.334612] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.353971] LustreError: 68553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.364021] LustreError: 68553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2063.434908] Lustre: server umount lustre-MDT0000 complete [ 2064.764307] LustreError: 69652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2068.459764] LustreError: 68550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2068.498688] LustreError: 68550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2072.929260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.079654] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2073.262180] Lustre: *** cfs_fail_loc=135, val=0*** [ 2073.390183] LustreError: 71652:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2073.396480] LustreError: 71652:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.105@tcp: cfg command failed: rc = -2 [ 2073.417138] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2073.417138] [ 2073.432803] LustreError: MGC192.168.203.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2073.442723] LustreError: 71640:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2073.447154] LustreError: 71640:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2073.463756] LustreError: 71640:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2073.476683] LustreError: 71640:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2073.519306] Lustre: server umount lustre-MDT0000 complete [ 2073.521254] LustreError: 71640:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2073.573512] LustreError: 68548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.585841] LustreError: 68548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2078.689944] LustreError: 68553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2078.707416] LustreError: 68553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2079.187231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.342976] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2079.596852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.676886] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2080.097705] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2083.448966] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2084.839212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2084.893045] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2084.969809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2089.965485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.976229] Lustre: Skipped 3 previous similar messages [ 2089.976284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2089.993338] Lustre: Skipped 1 previous similar message [ 2091.766198] Lustre: server umount lustre-OST0000 complete [ 2100.193677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.196607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2100.199311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.209954] Lustre: Skipped 1 previous similar message [ 2101.532355] Lustre: server umount lustre-MDT0000 complete [ 2104.744790] LustreError: 68536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827749 with bad export cookie 8156371024581902936 [ 2104.750842] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2104.754093] LustreError: 68536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2105.055103] Lustre: server umount lustre-MDT0001 complete [ 2114.945369] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2117.273733] Key type lgssc unregistered [ 2117.588266] LNet: 73432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.601949] LNetError: 73432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2117.618165] LNet: Removed LNI 192.168.203.105@tcp [ 2118.364920] Key type .llcrypt unregistered [ 2118.369280] Key type ._llcrypt unregistered [ 2138.411780] Key type ._llcrypt registered [ 2138.414633] Key type .llcrypt registered [ 2138.548565] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2139.447626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2139.487101] alg: No test for adler32 (adler32-zlib) [ 2140.482180] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 2140.700556] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2142.408617] Key type lgssc registered [ 2143.534987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2149.634554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2149.664153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2150.975142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.490192] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2160.110796] LustreError: 74824:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.124150] LustreError: 74824:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2161.186942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.643251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2165.937426] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2172.280906] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2176.745575] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2182.048641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2182.363077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.366516] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2195.364584] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2195.730569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2201.372722] Lustre: Failing over lustre-MDT0000 [ 2201.685523] Lustre: server umount lustre-MDT0000 complete [ 2202.492127] LustreError: 77375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2202.596710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2202.613156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.177151] LustreError: 74824:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.180057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.212578] LustreError: 74824:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2206.232331] Lustre: Skipped 1 previous similar message [ 2211.315817] LustreError: 74823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.348238] LustreError: 74823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2212.190236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.465101] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2212.695335] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2212.696018] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.5@tcp (not set up) [ 2212.824403] LustreError: 77910:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2212.830184] LustreError: 77910:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.105@tcp: cfg command failed: rc = -9 [ 2212.839704] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2212.839704] [ 2212.850054] LustreError: MGC192.168.203.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2212.864143] LustreError: 77898:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2212.882609] LustreError: 77898:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2212.892769] LustreError: 77898:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2212.900278] LustreError: 77898:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2212.927302] Lustre: server umount lustre-MDT0000 complete [ 2212.933911] LustreError: 77898:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2216.429558] LustreError: 77375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.461152] LustreError: 77375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2220.583885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.723540] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2220.735234] LustreError: 74815:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2221.203481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2221.277217] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2222.956231] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2225.313881] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2226.671263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2226.700661] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2226.753305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2231.783046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.788933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2231.791228] Lustre: Skipped 1 previous similar message [ 2231.798750] Lustre: Skipped 1 previous similar message [ 2233.812198] Lustre: server umount lustre-OST0000 complete [ 2236.897314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2236.897679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.923793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.933887] Lustre: Skipped 1 previous similar message [ 2242.017689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.028034] Lustre: Skipped 1 previous similar message [ 2247.137715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.145135] Lustre: Skipped 1 previous similar message [ 2251.231519] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2251.395869] Lustre: server umount lustre-MDT0000 complete [ 2254.316100] LustreError: 74803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827899 with bad export cookie 2655498153803836636 [ 2254.320852] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2254.327893] LustreError: 74803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2254.544919] Lustre: server umount lustre-MDT0001 complete [ 2262.924517] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2265.154051] Key type lgssc unregistered [ 2265.406689] LNet: 79694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2265.412593] LNetError: 79694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2265.424489] LNet: Removed LNI 192.168.203.105@tcp [ 2266.146613] Key type .llcrypt unregistered [ 2266.149036] Key type ._llcrypt unregistered [ 2277.331589] Key type ._llcrypt registered [ 2277.334805] Key type .llcrypt registered [ 2277.427502] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2277.754613] Key type .llcrypt unregistered [ 2277.756745] Key type ._llcrypt unregistered [ 2299.660679] Key type ._llcrypt registered [ 2299.663327] Key type .llcrypt registered [ 2299.781187] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2300.892895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2300.949859] alg: No test for adler32 (adler32-zlib) [ 2301.906467] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 2302.088682] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2303.745582] Key type lgssc registered [ 2305.038328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2312.484404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2312.505352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.934392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.765226] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2323.429205] LustreError: 81600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.443504] LustreError: 81600:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2326.367989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.851072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2332.488257] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2340.538575] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2346.804568] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2353.123515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2353.423683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.573486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2358.573659] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2364.690706] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2382.314588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2382.320121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.339776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.350506] Lustre: Skipped 1 previous similar message [ 2383.425502] Lustre: server umount lustre-OST0000 complete [ 2388.453088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2388.461393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.475718] Lustre: Skipped 1 previous similar message [ 2388.482350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2392.544833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2392.557367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.665388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2397.673733] Lustre: Skipped 2 previous similar messages [ 2402.271160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2402.461611] Lustre: server umount lustre-MDT0000 complete [ 2402.787015] LustreError: 82329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.811938] LustreError: 82329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2405.480039] LustreError: 81581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828050 with bad export cookie 10304040066602939321 [ 2405.485042] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2405.498840] LustreError: 81581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2405.901362] Lustre: server umount lustre-MDT0001 complete [ 2417.339183] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2419.564355] Key type lgssc unregistered [ 2419.812131] LNet: 85158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.820557] LNetError: 85158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.838469] LNet: Removed LNI 192.168.203.105@tcp [ 2420.712523] Key type .llcrypt unregistered [ 2420.714400] Key type ._llcrypt unregistered [ 2446.732922] Key type ._llcrypt registered [ 2446.739217] Key type .llcrypt registered [ 2446.902761] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2449.085374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2449.137752] alg: No test for adler32 (adler32-zlib) [ 2450.376961] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 2450.842690] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2452.623217] Key type lgssc registered [ 2454.630407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2463.886517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2463.909570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2465.350828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2469.786504] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2475.495050] LustreError: 86576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.513328] LustreError: 86576:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2478.534084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2478.976053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2482.965506] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2490.208708] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2495.937323] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2502.124409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2502.516213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.589277] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2510.714680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2514.446739] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2521.057323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2521.067268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2521.080390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2521.082126] Lustre: Skipped 1 previous similar message [ 2521.102433] Lustre: Skipped 1 previous similar message [ 2526.186385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2526.195960] Lustre: Skipped 1 previous similar message [ 2527.173462] Lustre: server umount lustre-OST0000 complete [ 2531.299424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2531.313068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2531.321762] Lustre: Skipped 1 previous similar message [ 2536.416553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2536.421121] Lustre: Skipped 1 previous similar message [ 2536.957361] Lustre: server umount lustre-MDT0000 complete [ 2540.247130] LustreError: 86557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828185 with bad export cookie 14793836317666418990 [ 2540.264761] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2540.502045] Lustre: server umount lustre-MDT0001 complete [ 2549.920816] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2551.960156] Key type lgssc unregistered [ 2552.200225] LNet: 90109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2552.208667] LNetError: 90109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2552.230747] LNet: Removed LNI 192.168.203.105@tcp [ 2553.102910] Key type .llcrypt unregistered [ 2553.108305] Key type ._llcrypt unregistered [ 2577.190832] Key type ._llcrypt registered [ 2577.195309] Key type .llcrypt registered [ 2577.347386] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2578.309477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2578.371133] alg: No test for adler32 (adler32-zlib) [ 2579.408602] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 2579.701129] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2581.367163] Key type lgssc registered [ 2582.312652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2590.315523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2590.336621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2591.712776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2596.699677] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2601.959234] LustreError: 91531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.981369] LustreError: 91531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2605.214995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.723622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2610.182824] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2616.827953] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2622.066321] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2629.423844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2629.726833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2636.909217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2637.440288] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2648.633783] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2659.820220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2659.828302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2659.846229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2660.172445] Lustre: server umount lustre-OST0000 complete [ 2666.978795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2667.000345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2667.020796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2668.002862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2673.121080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2673.132910] Lustre: Skipped 3 previous similar messages [ 2677.216369] Lustre: 90843:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786828306/real 1786828306] req@0000000081d6ca42 x1873625191837824/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786828322 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2677.253453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2677.270604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2678.239861] Lustre: 90840:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786828307/real 1786828307] req@0000000081a87332 x1873625191838592/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786828323 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2678.241737] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2678.521348] Lustre: server umount lustre-MDT0000 complete [ 2682.353199] LustreError: 91530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.372752] LustreError: 91530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2683.430135] Lustre: 90841:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786828312/real 1786828312] req@00000000c8dbf5a8 x1873625191840256/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786828328 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2683.511174] LustreError: 91511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828328 with bad export cookie 13419594078243679309 [ 2683.517797] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2683.525631] LustreError: 91511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2683.963825] Lustre: server umount lustre-MDT0001 complete [ 2697.438566] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2700.928906] Key type lgssc unregistered [ 2701.211547] LNet: 95066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2701.217830] LNetError: 95066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2701.237622] LNet: Removed LNI 192.168.203.105@tcp [ 2702.317556] Key type .llcrypt unregistered [ 2702.320625] Key type ._llcrypt unregistered [ 2730.494425] Key type ._llcrypt registered [ 2730.507244] Key type .llcrypt registered [ 2730.775600] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2732.506486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2732.541356] alg: No test for adler32 (adler32-zlib) [ 2733.786520] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 2734.202109] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2736.055153] Key type lgssc registered [ 2738.313932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2748.783309] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2748.798640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2764.319444] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2769.439490] LustreError: 96458:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 2796.063256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.756655] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2808.423396] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2813.803985] Lustre: server umount lustre-OST0000 complete [ 2823.057219] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 17:14:25 (1786828465) [ 2829.489404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.922980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2834.182963] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2835.438717] LustreError: 98098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.489900] LustreError: 98098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2839.591426] LustreError: 98097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.417746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.788863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2847.346626] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2854.829246] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2860.150329] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2867.587516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2868.084085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.561934] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2874.251401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2881.477426] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2897.889458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2897.913762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2897.916122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2897.942358] Lustre: Skipped 1 previous similar message [ 2897.966558] Lustre: Skipped 1 previous similar message [ 2902.264626] Lustre: server umount lustre-OST0000 complete [ 2908.129160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2908.133580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2908.157083] Lustre: Skipped 1 previous similar message [ 2908.169173] Lustre: Skipped 1 previous similar message [ 2912.637760] Lustre: server umount lustre-MDT0000 complete [ 2913.248964] LustreError: 98097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2913.266908] LustreError: 98097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2916.462480] LustreError: 98833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828561 with bad export cookie 12921255999941455561 [ 2916.474336] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2916.488902] LustreError: 98833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2917.121154] Lustre: server umount lustre-MDT0001 complete [ 2928.528769] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2930.845615] Key type lgssc unregistered [ 2931.095600] LNet: 101792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2931.111542] LNetError: 101792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2931.131771] LNet: Removed LNI 192.168.203.105@tcp [ 2932.264260] Key type .llcrypt unregistered [ 2932.267148] Key type ._llcrypt unregistered [ 2980.597580] Key type ._llcrypt registered [ 2980.599696] Key type .llcrypt registered [ 2980.720844] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2982.644337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2982.692344] alg: No test for adler32 (adler32-zlib) [ 2984.069469] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 2984.503889] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2986.295893] Key type lgssc registered [ 2988.439412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2996.913674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2996.954222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.393663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3002.753681] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3009.009589] LustreError: 103980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.034126] LustreError: 103980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3010.305635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.545541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3014.192519] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3021.950380] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3027.983214] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3035.622939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3036.311110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3043.116707] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3043.513350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3052.659350] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3066.345202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3066.345825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3066.355600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3066.409981] Lustre: Skipped 1 previous similar message [ 3071.461652] LustreError: 105803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3071.481811] LustreError: 105803:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3071.641492] Lustre: server umount lustre-OST0000 complete [ 3076.587476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.594477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3076.613917] Lustre: Skipped 1 previous similar message [ 3076.618538] Lustre: Skipped 1 previous similar message [ 3081.619429] Lustre: server umount lustre-MDT0000 complete [ 3081.719137] LustreError: 103975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3081.740424] LustreError: 103975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3085.418433] LustreError: 103961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828730 with bad export cookie 2259458086966835593 [ 3085.432481] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3085.435735] LustreError: 103961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3085.789664] Lustre: server umount lustre-MDT0001 complete [ 3095.198272] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 3099.114970] Key type lgssc unregistered [ 3099.413509] LNet: 107538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3099.424990] LNetError: 107538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3099.450744] LNet: Removed LNI 192.168.203.105@tcp [ 3100.407345] Key type .llcrypt unregistered [ 3100.409550] Key type ._llcrypt unregistered [ 3119.989614] Key type ._llcrypt registered [ 3119.992555] Key type .llcrypt registered [ 3120.079600] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3120.899078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3121.011446] alg: No test for adler32 (adler32-zlib) [ 3122.158511] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 3122.403580] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 3124.103163] Key type lgssc registered [ 3124.926744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3131.748341] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3131.768135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3147.423421] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3152.545242] LustreError: 108780:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 3179.360312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3187.077289] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3194.457655] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3201.211144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3201.660107] LustreError: MGC192.168.203.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3201.677228] LustreError: 109805:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3201.684097] LustreError: 109805:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3201.690069] LustreError: 109805:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3201.707922] LustreError: 108788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828846 with bad export cookie 8414652577906333797 [ 3201.731530] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3201.825933] Lustre: server umount lustre-MDT0000 complete [ 3201.829849] LustreError: 109805:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3210.458871] Lustre: server umount lustre-OST0000 complete [ 3220.602241] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 3230.329938] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3264.131278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.408773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3264.446137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3264.502299] Lustre: lustre-MDT0000: new disk, initializing [ 3264.595789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.609740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3269.236628] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3280.764981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3281.048897] Lustre: 112596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3281.170179] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3281.186152] Lustre: Skipped 1 previous similar message [ 3281.415730] Lustre: lustre-MDT0001: new disk, initializing [ 3281.662774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3281.741798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3281.767897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3287.447711] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3298.683947] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3306.197736] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3313.436265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3313.668034] Lustre: lustre-OST0000: new disk, initializing [ 3313.671638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3313.676891] Lustre: 114564:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3313.756608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3319.448716] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3322.958232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3322.978182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3323.155660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3328.719664] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3333.149222] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3333.502290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3337.763491] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3337.937502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3343.841281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3343.851720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.870281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3343.876137] Lustre: Skipped 1 previous similar message [ 3346.190637] Lustre: server umount lustre-OST0000 complete [ 3353.567982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3353.573901] LustreError: Skipped 1 previous similar message [ 3353.589494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.607510] Lustre: Skipped 1 previous similar message [ 3353.618776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3356.106717] Lustre: server umount lustre-MDT0000 complete [ 3361.071176] LustreError: 112587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829006 with bad export cookie 8414652577906333895 [ 3361.088455] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3361.530277] Lustre: server umount lustre-MDT0001 complete [ 3371.754573] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 17:23:34 (1786829014) [ 3375.219941] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3390.187652] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 3399.767604] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3435.521956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.801673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3435.836148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3435.909848] Lustre: lustre-MDT0000: new disk, initializing [ 3436.017187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3436.033254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3440.082888] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3453.018587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.109721] Lustre: 119467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3453.208691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3453.215542] Lustre: Skipped 1 previous similar message [ 3453.278324] Lustre: lustre-MDT0001: new disk, initializing [ 3453.406356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3453.452240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3453.465196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3461.305511] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3472.528517] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3478.358206] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3486.575605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3486.938239] Lustre: lustre-OST0000: new disk, initializing [ 3486.945339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3486.956445] Lustre: 121431:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3487.172646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3492.938908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3492.954508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3493.103560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3493.805615] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3503.740627] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3508.476236] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3508.634464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3513.135874] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3513.367365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3518.952151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3518.959661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3518.963928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3518.969789] Lustre: Skipped 2 previous similar messages [ 3518.990723] Lustre: Skipped 3 previous similar messages [ 3521.633552] Lustre: server umount lustre-OST0000 complete [ 3525.088106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3525.098060] LustreError: Skipped 1 previous similar message [ 3525.105926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.122772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3529.187323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.190191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3529.216949] Lustre: Skipped 1 previous similar message [ 3531.330225] Lustre: server umount lustre-MDT0000 complete [ 3534.304840] LustreError: 119473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3534.316258] LustreError: 119473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3535.045633] LustreError: 119459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829179 with bad export cookie 8414652577906334777 [ 3535.057715] LustreError: 119459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3535.060183] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3535.463841] Lustre: server umount lustre-MDT0001 complete [ 3546.461910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3547.101224] LustreError: 123650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3547.205869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3551.625133] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3552.226269] LustreError: 123649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3557.346095] LustreError: 123650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3560.836687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3565.660947] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3573.041788] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3578.360462] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3584.675565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3591.872305] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3601.008945] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3609.208253] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3616.737548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3616.742549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3616.761904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3620.912785] Lustre: server umount lustre-OST0000 complete [ 3628.001130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3628.008217] LustreError: Skipped 1 previous similar message [ 3630.936642] Lustre: server umount lustre-MDT0000 complete [ 3633.633927] LustreError: 123649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3633.680352] LustreError: 123649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3635.942591] LustreError: 123631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829280 with bad export cookie 8414652577906335659 [ 3635.955967] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3635.977780] LustreError: 123631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3636.428140] Lustre: server umount lustre-MDT0001 complete [ 3648.904388] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 3652.189579] Key type lgssc unregistered [ 3652.468929] LNet: 127282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3652.475051] LNetError: 127282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3652.489974] LNet: Removed LNI 192.168.203.105@tcp [ 3653.296496] Key type .llcrypt unregistered [ 3653.299057] Key type ._llcrypt unregistered [ 3673.451192] Key type ._llcrypt registered [ 3673.454218] Key type .llcrypt registered [ 3673.538182] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 3688.966093] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3689.919628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3689.949151] alg: No test for adler32 (adler32-zlib) [ 3690.968989] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 3691.227267] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 3692.935177] Key type lgssc registered [ 3693.953382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3725.710875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3725.736869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3726.930859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3726.956043] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3727.043891] Lustre: lustre-MDT0000: new disk, initializing [ 3727.105171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3727.120515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3731.472928] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3741.335677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3741.491637] Lustre: 130436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3741.586670] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3741.592080] Lustre: Skipped 1 previous similar message [ 3741.690243] Lustre: lustre-MDT0001: new disk, initializing [ 3741.832675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3741.887326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3741.907202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3746.679670] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3758.338078] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3766.331749] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3773.745800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3774.047888] Lustre: lustre-OST0000: new disk, initializing [ 3774.051362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3774.057150] Lustre: 132409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3774.152870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3780.905922] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3783.305927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3783.327329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3783.438787] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3789.812063] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3793.386508] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3793.626957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3797.500419] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3797.766913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3804.134764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3804.140951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3804.146634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3804.162281] Lustre: Skipped 1 previous similar message [ 3804.189466] LustreError: Skipped 1 previous similar message [ 3805.960096] Lustre: server umount lustre-OST0000 complete [ 3813.349846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3813.362842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3813.378114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3813.382259] Lustre: Skipped 1 previous similar message [ 3814.371487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3814.384235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3814.391484] Lustre: Skipped 1 previous similar message [ 3816.686714] Lustre: server umount lustre-MDT0000 complete [ 3820.518373] LustreError: 132408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829465 with bad export cookie 10486606995904133487 [ 3820.522529] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3820.532475] LustreError: 132408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3821.064980] Lustre: server umount lustre-MDT0001 complete [ 3830.962497] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 17:31:13 (1786829473) [ 3838.414891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3838.734623] LustreError: 134820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3838.756128] LustreError: 134820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3838.838327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3842.883866] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3844.064076] LustreError: 134821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.171749] LustreError: 134820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3853.283480] LustreError: 134821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3854.066368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3854.323239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3859.423806] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3868.164890] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3874.239370] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3879.907738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3879.907780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3879.910605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3879.951855] Lustre: Skipped 1 previous similar message [ 3885.029099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3885.039177] Lustre: Skipped 1 previous similar message [ 3890.143135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3890.375079] Lustre: server umount lustre-MDT0000 complete [ 3894.270261] LustreError: 134801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829539 with bad export cookie 10486606995904134362 [ 3894.275936] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3894.286035] LustreError: 134801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3894.635590] Lustre: server umount lustre-MDT0001 complete [ 3902.668530] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 17:32:25 (1786829545) [ 3910.019473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3925.856564] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3930.975512] LustreError: 137234:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 3956.703313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3963.500915] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3970.736600] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3972.949930] Lustre: server umount lustre-OST0000 complete [ 3980.337244] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 17:33:43 (1786829623) [ 3986.414565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.799423] LustreError: 138680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3986.862276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3990.652242] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3996.141338] LustreError: 138680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3996.160176] LustreError: 138680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3998.773200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4002.993190] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4009.949385] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4015.614891] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4021.443308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4021.896826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4021.903557] Lustre: Skipped 1 previous similar message [ 4027.017082] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4033.997664] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4041.698954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4041.702618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4041.717578] Lustre: Skipped 3 previous similar messages [ 4041.733755] Lustre: Skipped 4 previous similar messages [ 4046.774809] Lustre: server umount lustre-MDT0000 complete [ 4046.819969] LustreError: 139409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4046.847372] LustreError: 139409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4050.142280] LustreError: 138660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829695 with bad export cookie 10486606995904135034 [ 4050.148380] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4050.156636] LustreError: 138660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4050.458215] Lustre: server umount lustre-MDT0001 complete [ 4064.055921] Lustre: server umount lustre-OST0000 complete [ 4072.230975] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 17:35:15 (1786829715) [ 4078.287479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4082.114847] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4084.192347] LustreError: 142332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4084.215293] LustreError: 142332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4089.946838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.358475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4090.362361] Lustre: Skipped 1 previous similar message [ 4094.276936] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4101.827416] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4108.119469] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4114.442929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4121.083446] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4123.158356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4127.635459] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4132.360907] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4132.595582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4136.444120] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4136.641432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4138.469296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4138.475654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4138.485181] Lustre: Skipped 1 previous similar message [ 4138.488776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4138.497663] Lustre: Skipped 1 previous similar message [ 4144.291500] Lustre: server umount lustre-OST0000 complete [ 4153.929249] Lustre: server umount lustre-MDT0000 complete [ 4156.990097] LustreError: 142313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829801 with bad export cookie 10486606995904136105 [ 4156.992520] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4157.000656] LustreError: 142313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4157.395735] Lustre: server umount lustre-MDT0001 complete [ 4164.755965] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 17:36:47 (1786829807) [ 4170.415980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4185.952131] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4191.071498] LustreError: 146488:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 4222.356390] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4228.769342] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4234.260837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4234.628458] LustreError: 146513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.647217] LustreError: 146513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4234.700230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4234.708207] Lustre: Skipped 2 previous similar messages [ 4234.766550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4239.010777] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4247.644516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4251.560281] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4259.200573] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4266.998709] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4272.208597] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4272.624063] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4277.186645] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4277.353682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4282.852540] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4282.868388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4282.870572] Lustre: Skipped 3 previous similar messages [ 4282.889610] Lustre: Skipped 8 previous similar messages [ 4285.199352] Lustre: server umount lustre-MDT0000 complete [ 4289.528686] LustreError: 146494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829934 with bad export cookie 10486606995904137127 [ 4289.550543] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4289.897899] Lustre: server umount lustre-MDT0001 complete [ 4304.252470] Lustre: server umount lustre-OST0000 complete [ 4313.254093] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 17:39:16 (1786829956) [ 4320.418524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4336.032270] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4341.151724] LustreError: 150682:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 4374.001159] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4383.147693] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4389.791064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4390.270946] LustreError: 150707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4390.292848] LustreError: 150707:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4390.528322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4394.153420] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4401.998062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4406.087773] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4413.146281] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4419.388730] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4426.043955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4426.187365] Lustre: lustre-OST0001: new disk, initializing [ 4426.190595] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4426.196273] Lustre: 153554:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4433.246926] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4433.998632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4434.012603] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4434.180205] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4443.499392] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4447.591713] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4447.937331] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4454.558718] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4454.925811] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4458.980468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4458.991143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4459.005979] Lustre: Skipped 1 previous similar message [ 4459.013218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4463.464434] Lustre: server umount lustre-OST0000 complete [ 4470.239991] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4470.250112] LustreError: Skipped 1 previous similar message [ 4473.149699] Lustre: server umount lustre-OST0001 complete [ 4483.260160] Lustre: server umount lustre-MDT0000 complete [ 4486.806315] LustreError: 150688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830131 with bad export cookie 10486606995904138079 [ 4486.817174] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4486.822684] LustreError: 150688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4487.300635] Lustre: server umount lustre-MDT0001 complete [ 4509.400300] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 17:42:32 (1786830152) [ 4510.850243] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4512.561679] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4514.418478] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 17:42:37 (1786830157) [ 4520.427543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4520.636330] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4520.659695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4520.816192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4520.824945] Lustre: Skipped 5 previous similar messages [ 4524.768087] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4531.929678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.995850] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4532.019851] Lustre: 157113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4537.066649] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4544.161376] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4550.239241] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4556.568828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4556.805297] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4562.626622] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4567.124282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4568.729161] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4574.199821] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4574.693494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4579.908294] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4580.226421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4582.371199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.395255] Lustre: Skipped 6 previous similar messages [ 4587.492563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4587.506114] Lustre: Skipped 10 previous similar messages [ 4588.435411] Lustre: server umount lustre-OST0000 complete [ 4610.590598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4613.102729] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4613.119595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4613.126210] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4613.161885] Lustre: Skipped 1 previous similar message [ 4617.725638] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4627.051677] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4632.945838] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4633.214192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4637.002041] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4637.305763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4642.700519] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8ddc09acf800.ost_server_uuid 50 [ 4644.073957] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8ddc09acf800.ost_server_uuid in FULL state after 0 sec [ 4644.332379] LustreError: 162147:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.5@tcp) returned error from blocking AST (req@00000000b6a082d0 x1873626358084352 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000ba00559e/0x9187e98099fe14b9 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.5@tcp remote: 0x5bbd3f8570117bc6 expref: 5 pid: 157119 timeout: 4744 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4644.349169] LustreError: lustre-MDT0000: A client on nid 192.168.203.5@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4644.354824] LustreError: 157109:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.5@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000ba00559e/0x9187e98099fe14b9 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.5@tcp remote: 0x5bbd3f8570117bc6 expref: 6 pid: 157119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4648.957427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4648.975048] LustreError: Skipped 2 previous similar messages [ 4654.058144] LustreError: 160736:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4654.077110] LustreError: 160736:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4654.168313] Lustre: server umount lustre-OST0000 complete [ 4664.358406] Lustre: server umount lustre-MDT0000 complete [ 4668.732109] LustreError: 157105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830313 with bad export cookie 10486606995904139262 [ 4668.745854] LustreError: 157105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4668.753210] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4683.231501] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4683.540595] Lustre: server umount lustre-MDT0001 complete [ 4697.773797] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4700.963882] Key type lgssc unregistered [ 4701.257618] LNet: 163228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4701.270993] LNetError: 163228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4702.310756] LNet: Removed LNI 192.168.203.105@tcp [ 4703.237580] Key type .llcrypt unregistered [ 4703.245833] Key type ._llcrypt unregistered [ 4729.197751] Key type ._llcrypt registered [ 4729.200559] Key type .llcrypt registered [ 4729.285717] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4730.779906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4730.838141] alg: No test for adler32 (adler32-zlib) [ 4731.908342] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 4732.228051] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4733.967159] Key type lgssc registered [ 4735.403565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4745.286551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4745.344942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4747.161455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4753.141021] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4757.472454] LustreError: 164646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4757.517685] LustreError: 164646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4762.593061] LustreError: 164645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4763.639880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.089047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4768.350839] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4776.128359] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4781.971656] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4788.687883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4789.073334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4792.177425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4795.575945] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4802.872291] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4811.567903] Lustre: Failing over lustre-MDT0000 [ 4811.991390] Lustre: server umount lustre-MDT0000 complete [ 4813.817922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.824217] LustreError: 164641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4813.851709] Lustre: Skipped 2 previous similar messages [ 4813.912135] LustreError: 164641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4818.914848] LustreError: 164642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4818.956469] LustreError: 164642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4824.039923] LustreError: 165690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4824.066290] LustreError: 165690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4824.370132] LustreError: 164625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.5@tcp arrived at 1786830469 with bad export cookie 5084995706880550032 [ 4829.154699] Lustre: 163957:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830458/real 1786830458] req@000000004ec87437 x1873627449421696/t0(0) o400->MGC192.168.203.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786830474 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4829.187621] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4832.493913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.275132] LustreError: 166461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4834.293973] LustreError: 166461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4839.794481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4839.872172] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4844.369232] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4845.040851] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4845.053830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4847.498960] Lustre: lustre-MDT0000: Denying connection for new client 2a835dce-d197-468f-b25b-888acffe24d3 (at 192.168.203.5@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 4855.265444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4855.277043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4855.286466] Lustre: Skipped 1 previous similar message [ 4860.387302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4860.404437] Lustre: Skipped 1 previous similar message [ 4861.347067] Lustre: server umount lustre-OST0000 complete [ 4865.032695] LustreError: 168376:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4865.039833] Lustre: 167725:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4865.047926] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4865.066139] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4865.084493] LustreError: 167725:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000092314ff8 x1873627449440512/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4865.114290] LustreError: 167725:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4865.124436] LustreError: 167725:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4865.133745] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 4865.508695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4865.532697] Lustre: Skipped 1 previous similar message [ 4865.549568] LustreError: 164641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4865.584146] LustreError: 164641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4865.664439] Lustre: server umount lustre-MDT0000 complete [ 4869.985172] LustreError: 164625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830514 with bad export cookie 5084995706880550137 [ 4869.994911] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4869.997523] LustreError: 163953:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4870.010578] LustreError: 164625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4872.778190] Lustre: server umount lustre-MDT0001 complete [ 4884.429890] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4887.199740] Key type lgssc unregistered [ 4887.502831] LNet: 169133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4887.513994] LNetError: 169133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4887.541604] LNet: Removed LNI 192.168.203.105@tcp [ 4888.328409] Key type .llcrypt unregistered [ 4888.329926] Key type ._llcrypt unregistered [ 4912.745738] Key type ._llcrypt registered [ 4912.748818] Key type .llcrypt registered [ 4912.849327] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4913.775605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4913.916098] alg: No test for adler32 (adler32-zlib) [ 4915.064497] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 4915.342294] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4917.055161] Key type lgssc registered [ 4918.352970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4925.228993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4925.242930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4926.825753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4931.592919] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4936.170946] LustreError: 170548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4936.226063] LustreError: 170548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4941.281237] LustreError: 170547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.564075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4942.052086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4946.939434] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4954.268788] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4960.464787] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4966.847888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4967.161406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4971.264303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4972.442517] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4979.530216] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4986.467549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.473523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4986.502409] Lustre: Skipped 1 previous similar message [ 4991.926555] Lustre: server umount lustre-OST0000 complete [ 4997.089371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4997.102995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4997.112538] Lustre: Skipped 1 previous similar message [ 5001.795100] Lustre: server umount lustre-MDT0000 complete [ 5002.208702] LustreError: 170548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5002.238659] LustreError: 170548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5004.865098] LustreError: 170528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830649 with bad export cookie 17186434122137198273 [ 5004.869970] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5004.884381] LustreError: 170528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5005.129601] Lustre: server umount lustre-MDT0001 complete [ 5015.228174] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5017.486072] Key type lgssc unregistered [ 5017.766673] LNet: 174084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5017.773687] LNetError: 174084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5017.787121] LNet: Removed LNI 192.168.203.105@tcp [ 5018.529850] Key type .llcrypt unregistered [ 5018.531931] Key type ._llcrypt unregistered [ 5046.996076] Key type ._llcrypt registered [ 5047.002260] Key type .llcrypt registered [ 5047.079338] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5047.854067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5047.893683] alg: No test for adler32 (adler32-zlib) [ 5048.881181] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 5049.154575] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5050.847152] Key type lgssc registered [ 5051.852242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5059.367143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5059.384780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5060.815685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.813134] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5070.313901] LustreError: 175781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5070.329127] LustreError: 175781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5073.501404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.240106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5078.445864] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5087.951846] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5096.107902] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5105.404288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5105.998254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5107.068354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5113.491059] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5121.570987] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5129.700399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5129.719365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5129.730386] Lustre: Skipped 1 previous similar message [ 5134.449231] Lustre: server umount lustre-OST0000 complete [ 5139.942113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.944446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5139.959405] Lustre: Skipped 2 previous similar messages [ 5139.974208] Lustre: Skipped 1 previous similar message [ 5145.059630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5145.072310] Lustre: Skipped 1 previous similar message [ 5149.161700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5152.223567] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5152.457826] Lustre: server umount lustre-MDT0000 complete [ 5154.272945] LustreError: 175777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5154.284606] LustreError: 175777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5156.651037] LustreError: 177625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830801 with bad export cookie 730873259808186928 [ 5156.667930] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5156.674047] LustreError: 177625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5157.074713] Lustre: server umount lustre-MDT0001 complete [ 5166.880712] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5169.758245] Key type lgssc unregistered [ 5169.981077] LNet: 179318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5169.985574] LNetError: 179318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5169.998377] LNet: Removed LNI 192.168.203.105@tcp [ 5170.673711] Key type .llcrypt unregistered [ 5170.680726] Key type ._llcrypt unregistered [ 5193.974410] Key type ._llcrypt registered [ 5193.977391] Key type .llcrypt registered [ 5194.082452] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5195.381663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5195.441778] alg: No test for adler32 (adler32-zlib) [ 5196.622792] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 5196.930812] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5198.714353] Key type lgssc registered [ 5199.511801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5211.747562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5211.777914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5213.486350] Lustre: *** cfs_fail_loc=135, val=0*** [ 5213.754294] LustreError: 180843:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5213.765092] LustreError: 180843:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.105@tcp: cfg command failed: rc = -2 [ 5213.775989] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5213.775989] [ 5213.786055] LustreError: MGC192.168.203.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5213.797387] LustreError: 180804:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5213.818828] LustreError: 180804:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5213.836896] LustreError: 180804:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5213.844223] LustreError: 180804:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5213.953223] Lustre: server umount lustre-MDT0000 complete [ 5213.961945] LustreError: 180804:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5221.273641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5221.762033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5225.892915] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5231.087636] LustreError: 181215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5231.132600] LustreError: 181215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5235.267559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5235.721792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5241.029678] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5247.490238] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5253.072717] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5256.160845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5256.165654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5256.174112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5257.184809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5257.191578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.326181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5260.342024] Lustre: Skipped 2 previous similar messages [ 5261.267132] Lustre: server umount lustre-MDT0000 complete [ 5264.567080] LustreError: 181194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830909 with bad export cookie 11898081813009525853 [ 5264.584213] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5264.849407] Lustre: server umount lustre-MDT0001 complete [ 5275.544444] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5278.137326] Key type lgssc unregistered [ 5278.421845] LNet: 183538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5278.428962] LNetError: 183538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5278.442513] LNet: Removed LNI 192.168.203.105@tcp [ 5279.269193] Key type .llcrypt unregistered [ 5279.271254] Key type ._llcrypt unregistered [ 5299.850931] Key type ._llcrypt registered [ 5299.859194] Key type .llcrypt registered [ 5299.985548] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5300.302777] Key type .llcrypt unregistered [ 5300.305316] Key type ._llcrypt unregistered [ 5316.999499] Key type ._llcrypt registered [ 5317.004685] Key type .llcrypt registered [ 5317.120498] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5318.595434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5318.624387] alg: No test for adler32 (adler32-zlib) [ 5319.687753] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 5319.940395] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5321.719717] Key type lgssc registered [ 5323.030934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5331.080707] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5331.109970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5346.720203] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5351.840715] LustreError: 185673:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 5378.527281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5383.810301] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5390.405289] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5397.008980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5397.567722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5397.644482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5401.730699] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5402.609528] LustreError: 185698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5402.649148] LustreError: 185698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5407.724813] LustreError: 185699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5409.773514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.121123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5413.672851] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5420.939238] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5426.325669] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5429.137516] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5438.438601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5438.447824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.469943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5438.953337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.984744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5443.862528] Lustre: server umount lustre-OST0000 complete [ 5449.185738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5449.203519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5454.308462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5454.317030] Lustre: Skipped 2 previous similar messages [ 5455.140951] Lustre: server umount lustre-MDT0000 complete [ 5458.404227] LustreError: 186712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5458.454434] LustreError: 186712:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5459.916294] LustreError: 185680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831104 with bad export cookie 8187628816238772652 [ 5459.928720] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5459.932261] LustreError: 185680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5460.926267] Lustre: server umount lustre-MDT0001 complete [ 5472.209586] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5475.493861] Key type lgssc unregistered [ 5475.863759] LNet: 189592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5475.874633] LNetError: 189592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5475.909796] LNet: Removed LNI 192.168.203.105@tcp [ 5476.975849] Key type .llcrypt unregistered [ 5476.979585] Key type ._llcrypt unregistered [ 5499.544268] Key type ._llcrypt registered [ 5499.545793] Key type .llcrypt registered [ 5499.612262] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5500.633827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5500.660763] alg: No test for adler32 (adler32-zlib) [ 5501.650639] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 5501.837034] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5503.479646] Key type lgssc registered [ 5504.790324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5511.988333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5512.027274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5513.473365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5518.139486] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5523.945290] LustreError: 191003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5523.957417] LustreError: 191003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5526.588214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.092914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5531.003660] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5537.687770] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5543.113839] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5550.415845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5550.731871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5556.669916] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5558.968960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5565.554096] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5571.426108] Lustre: Failing over lustre-MDT0000 [ 5572.047196] Lustre: server umount lustre-MDT0000 complete [ 5573.093420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5573.111843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.136034] LustreError: 191004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5574.636120] LustreError: 191003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5574.645069] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5574.664558] LustreError: 191003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5574.726086] Lustre: Skipped 1 previous similar message [ 5578.583981] LustreError: 191000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5578.615712] LustreError: 191000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5583.710586] LustreError: 190998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.736754] LustreError: 190998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5591.071652] Lustre: 190316:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786831219/real 1786831219] req@00000000e2eb3924 x1873628255775744/t0(0) o400->MGC192.168.203.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786831235 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5591.123596] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5592.111589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5593.953190] LustreError: 192054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5594.002104] LustreError: 192054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5601.766227] LustreError: 190314:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000071cdba8e x1873628255781760/t0(0) o250->MGC192.168.203.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5602.179477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5602.215302] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5604.186194] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5606.179560] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5607.411615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5607.435731] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5607.490783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5610.754270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5614.967264] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5622.763829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5622.778080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5622.800490] Lustre: Skipped 1 previous similar message [ 5626.975118] Lustre: server umount lustre-OST0000 complete [ 5632.997605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5633.000319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5633.002381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5633.008512] Lustre: Skipped 1 previous similar message [ 5633.014069] Lustre: Skipped 2 previous similar messages [ 5638.112133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5638.116261] Lustre: Skipped 1 previous similar message [ 5642.209323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5642.235482] Lustre: Skipped 1 previous similar message [ 5644.255143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5644.517367] Lustre: server umount lustre-MDT0000 complete [ 5647.336105] LustreError: 190999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5647.372631] LustreError: 190999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5650.302846] LustreError: 190984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831295 with bad export cookie 14351404334557091846 [ 5650.317431] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5650.340384] LustreError: 190984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5650.958274] Lustre: server umount lustre-MDT0001 complete [ 5663.340913] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5665.777792] Key type lgssc unregistered [ 5666.024710] LNet: 195810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5666.029653] LNetError: 195810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5666.044084] LNet: Removed LNI 192.168.203.105@tcp [ 5666.923196] Key type .llcrypt unregistered [ 5666.925834] Key type ._llcrypt unregistered [ 5690.250973] Key type ._llcrypt registered [ 5690.253109] Key type .llcrypt registered [ 5690.359770] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5691.271795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5691.358446] alg: No test for adler32 (adler32-zlib) [ 5692.538343] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 5692.777099] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5694.431188] Key type lgssc registered [ 5695.521285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5702.384749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5702.401917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.981526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5708.215818] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5713.378122] LustreError: 197224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5713.403955] LustreError: 197224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5716.261720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5716.628342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5720.595671] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5726.908654] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5732.066302] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5738.593405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5738.991206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5745.793911] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5748.212382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5754.713481] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5758.619149] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5765.978800] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5779.642503] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5793.253295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5793.259794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5793.271799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5794.280952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5794.281164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5794.312844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5798.756372] Lustre: server umount lustre-OST0000 complete [ 5803.488237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5803.493829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5803.518032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5803.530783] Lustre: Skipped 1 previous similar message [ 5808.592581] Lustre: server umount lustre-MDT0000 complete [ 5809.636903] LustreError: 197223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5809.656207] LustreError: 197223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5812.440356] LustreError: 197204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831457 with bad export cookie 16250660868529329751 [ 5812.452238] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5812.457387] LustreError: 197204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5812.819952] Lustre: server umount lustre-MDT0001 complete [ 5823.256389] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5825.968639] Key type lgssc unregistered [ 5826.259202] LNet: 201099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5826.266265] LNetError: 201099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5826.287748] LNet: Removed LNI 192.168.203.105@tcp [ 5827.215842] Key type .llcrypt unregistered [ 5827.218781] Key type ._llcrypt unregistered [ 5847.335394] Key type ._llcrypt registered [ 5847.337605] Key type .llcrypt registered [ 5847.470527] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5848.439655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5848.583260] alg: No test for adler32 (adler32-zlib) [ 5849.940358] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 5850.366283] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5852.095152] Key type lgssc registered [ 5853.505521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5861.371479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5861.389619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5862.899390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5867.702481] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5872.103483] LustreError: 202364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5872.135508] LustreError: 202364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5875.437674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5875.762830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5880.250315] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5887.760460] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5893.707643] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5901.216121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5901.744673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5907.889697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5908.341856] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5915.349537] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5921.427693] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5932.347242] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5944.537974] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5948.149091] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5956.070920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5956.076509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5956.095927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5961.185955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5961.200533] Lustre: Skipped 1 previous similar message [ 5961.621699] Lustre: server umount lustre-OST0000 complete [ 5966.311113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.311785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5966.333886] Lustre: Skipped 2 previous similar messages [ 5966.342457] Lustre: Skipped 1 previous similar message [ 5971.220921] Lustre: server umount lustre-MDT0000 complete [ 5971.424317] LustreError: 202359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5971.437768] LustreError: 202359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5974.528462] LustreError: 202345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831619 with bad export cookie 2538937158587453463 [ 5974.535405] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5974.540818] LustreError: 202345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5974.987771] Lustre: server umount lustre-MDT0001 complete [ 5985.735946] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5988.089531] Key type lgssc unregistered [ 5988.316864] LNet: 207291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5988.323478] LNetError: 207291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5988.338658] LNet: Removed LNI 192.168.203.105@tcp [ 5989.096990] Key type .llcrypt unregistered [ 5989.099845] Key type ._llcrypt unregistered [ 6011.750177] Key type ._llcrypt registered [ 6011.753945] Key type .llcrypt registered [ 6011.957537] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6012.857212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6012.869414] alg: No test for adler32 (adler32-zlib) [ 6013.891786] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 6014.045921] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 6015.671187] Key type lgssc registered [ 6017.074778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6024.462228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6024.483712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.760789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6029.930843] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6035.943872] LustreError: 208702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.967066] LustreError: 208702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6037.994665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6038.313973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6042.230618] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6049.113489] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6054.581444] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6061.039526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6061.386381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6067.252456] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6070.653183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6074.470665] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6089.184875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6089.196962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6089.219875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6091.246756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6091.247020] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6091.276109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6095.210074] Lustre: server umount lustre-OST0000 complete [ 6099.936537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6099.945547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6099.979446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6099.988067] Lustre: Skipped 1 previous similar message [ 6105.556625] Lustre: server umount lustre-MDT0000 complete [ 6106.594523] LustreError: 208699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6106.602819] LustreError: 208699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6109.215643] LustreError: 208684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831754 with bad export cookie 7615747719882660703 [ 6109.224668] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6109.225777] LustreError: 208684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6109.670136] Lustre: server umount lustre-MDT0001 complete [ 6122.068674] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 6125.633942] Key type lgssc unregistered [ 6125.958831] LNet: 212387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6125.969272] LNetError: 212387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6125.996270] LNet: Removed LNI 192.168.203.105@tcp [ 6126.900127] Key type .llcrypt unregistered [ 6126.901729] Key type ._llcrypt unregistered [ 6149.397630] Key type ._llcrypt registered [ 6149.400177] Key type .llcrypt registered [ 6149.527904] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6150.538928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6150.573241] alg: No test for adler32 (adler32-zlib) [ 6151.583331] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 6151.771261] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 6153.455142] Key type lgssc registered [ 6154.454565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6162.682297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6162.692796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6164.078374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6167.766089] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6173.682278] LustreError: 213657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6173.691500] LustreError: 213657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6175.518406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6175.998248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6180.199978] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6186.662991] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6192.070149] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6198.371588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6198.818170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6203.750615] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6209.078627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6210.420134] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6224.366535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6224.368582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6224.378202] Lustre: Skipped 1 previous similar message [ 6224.378976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6228.269553] Lustre: server umount lustre-OST0000 complete [ 6234.597426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6234.604022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6234.622484] Lustre: Skipped 1 previous similar message [ 6234.661156] Lustre: Skipped 2 previous similar messages [ 6238.912498] Lustre: server umount lustre-MDT0000 complete [ 6239.728268] LustreError: 213657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6239.759848] LustreError: 213657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6242.527420] LustreError: 213639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831887 with bad export cookie 895094166377754963 [ 6242.537221] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6242.537278] LustreError: 213639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6242.950870] Lustre: server umount lustre-MDT0001 complete [ 6257.321816] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 6261.114022] Key type lgssc unregistered [ 6261.384496] LNet: 217346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6261.410838] LNetError: 217346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6261.427236] LNet: Removed LNI 192.168.203.105@tcp [ 6262.199183] Key type .llcrypt unregistered [ 6262.201376] Key type ._llcrypt unregistered [ 6278.349691] Key type ._llcrypt registered [ 6278.351917] Key type .llcrypt registered [ 6278.436349] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6279.687111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6279.720495] alg: No test for adler32 (adler32-zlib) [ 6280.710564] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 6280.929459] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 6282.623353] Key type lgssc registered [ 6283.633096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6289.546511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6289.565804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6290.971548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6295.615732] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6300.139434] LustreError: 218598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6300.153669] LustreError: 218598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6303.125815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6303.417458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6306.818530] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6313.996545] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6319.229984] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6325.067807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6325.524844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6330.480436] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6335.789332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6337.927420] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6346.422932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6346.473230] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6346.714682] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6352.936972] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6361.277527] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6363.062273] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6375.936093] Lustre: Permanently deactivating lustre-OST0001 [ 6375.952763] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6381.667442] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6388.241465] Lustre: Permanently reactivating lustre-OST0001 [ 6388.272534] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6388.277410] Lustre: Skipped 2 previous similar messages [ 6388.920994] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6388.940563] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6388.949271] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6388.960094] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6389.054475] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6402.540770] Lustre: server umount lustre-OST0001 complete [ 6403.053381] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.060410] LustreError: 220419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6403.084966] Lustre: Skipped 2 previous similar messages [ 6403.150895] LustreError: 220419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6407.648293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6407.658661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6407.678783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6408.160718] LustreError: 220420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6408.191036] LustreError: 220420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6413.284833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6413.285807] LustreError: 221041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6413.295132] Lustre: Skipped 3 previous similar messages [ 6413.341316] LustreError: 221041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6413.978122] Lustre: server umount lustre-OST0000 complete [ 6418.401845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6418.409553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6418.425078] Lustre: Skipped 2 previous similar messages [ 6418.453276] Lustre: Skipped 1 previous similar message [ 6423.520977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6423.525012] Lustre: Skipped 1 previous similar message [ 6428.643951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6428.667190] Lustre: Skipped 1 previous similar message [ 6432.225143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6432.486723] Lustre: server umount lustre-MDT0000 complete [ 6433.762708] LustreError: 222239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6433.790936] LustreError: 222239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6436.712663] LustreError: 222380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832081 with bad export cookie 13080822046345888436 [ 6436.723033] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6437.290993] Lustre: server umount lustre-MDT0001 complete [ 6447.438814] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 6450.146152] Key type lgssc unregistered [ 6450.566351] LNet: 223647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6450.576844] LNetError: 223647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6450.600932] LNet: Removed LNI 192.168.203.105@tcp [ 6451.622965] Key type .llcrypt unregistered [ 6451.625140] Key type ._llcrypt unregistered [ 6489.572628] Key type ._llcrypt registered [ 6489.574523] Key type .llcrypt registered [ 6489.664750] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6490.317918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6490.422841] alg: No test for adler32 (adler32-zlib) [ 6491.477933] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 6491.708087] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 6493.343171] Key type lgssc registered [ 6494.848267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6503.299868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6503.315652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6504.652421] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6504.704623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6504.873268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6509.643667] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6518.128901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6518.261823] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6518.297575] Lustre: 225832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6518.730538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6524.835936] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6534.529681] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6540.580539] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6547.741258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6548.057823] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6548.342652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6554.261167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6554.646173] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6562.008897] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6566.208219] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6572.519190] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6577.990750] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6586.893374] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6599.750944] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6599.755472] Lustre: Skipped 1 previous similar message [ 6621.972758] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6621.978138] Lustre: Skipped 2 previous similar messages [ 6657.040324] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6657.050468] Lustre: Skipped 3 previous similar messages [ 6733.554483] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6733.565587] Lustre: Skipped 8 previous similar messages [ 6743.520974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6743.532743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6743.553829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6744.032897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6744.045510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6744.071644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6744.081363] Lustre: Skipped 1 previous similar message [ 6747.341350] Lustre: server umount lustre-OST0000 complete [ 6754.273351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6754.275870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6754.285783] Lustre: Skipped 1 previous similar message [ 6754.308533] Lustre: Skipped 1 previous similar message [ 6758.942887] Lustre: server umount lustre-MDT0000 complete [ 6762.923725] LustreError: 228402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832407 with bad export cookie 1910012079302894188 [ 6762.925549] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6762.938586] LustreError: 228402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6763.427737] Lustre: server umount lustre-MDT0001 complete [ 6772.916100] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 6776.169954] Key type lgssc unregistered [ 6776.483870] LNet: 230649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6776.489937] LNetError: 230649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6776.502203] LNet: Removed LNI 192.168.203.105@tcp [ 6777.265176] Key type .llcrypt unregistered [ 6777.266828] Key type ._llcrypt unregistered [ 6803.366138] Key type ._llcrypt registered [ 6803.367918] Key type .llcrypt registered [ 6803.448209] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6804.312347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6804.431423] alg: No test for adler32 (adler32-zlib) [ 6805.423705] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 6805.627150] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 6807.295153] Key type lgssc registered [ 6808.208757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6815.266489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6815.284748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6816.766502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6820.889805] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6826.976742] LustreError: 232062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6826.995376] LustreError: 232062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6827.948953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6828.233963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6833.375052] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6840.273531] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6846.022793] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6851.663560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6851.922440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6857.990627] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6861.165280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6868.164760] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6884.833778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6884.847087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6884.863862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6886.882835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6886.883767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6886.885941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6886.901739] Lustre: Skipped 1 previous similar message [ 6890.441541] Lustre: server umount lustre-OST0000 complete [ 6895.072406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6895.072583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6895.088609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6895.094533] Lustre: Skipped 1 previous similar message [ 6897.120065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6897.131897] Lustre: Skipped 2 previous similar messages [ 6902.243163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6902.248884] Lustre: Skipped 1 previous similar message [ 6908.386435] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6908.598396] Lustre: server umount lustre-MDT0000 complete [ 6911.457509] LustreError: 234026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6911.484698] LustreError: 234026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6911.911580] LustreError: 232042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832556 with bad export cookie 1721420984177896503 [ 6911.917936] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6911.936264] LustreError: 232042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6912.416787] Lustre: server umount lustre-MDT0001 complete [ 6921.693589] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 6924.072597] Key type lgssc unregistered [ 6924.342470] LNet: 235757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6924.360700] LNetError: 235757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6924.383830] LNet: Removed LNI 192.168.203.105@tcp [ 6925.624507] Key type .llcrypt unregistered [ 6925.628796] Key type ._llcrypt unregistered [ 6948.120255] Key type ._llcrypt registered [ 6948.123795] Key type .llcrypt registered [ 6948.290926] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6949.702298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6949.753269] alg: No test for adler32 (adler32-zlib) [ 6951.111395] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 6951.581945] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 6953.375737] Key type lgssc registered [ 6954.842534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7226.556903] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 7228.799466] Key type lgssc unregistered [ 7229.086514] LNet: 237784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7229.107172] LNetError: 237784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7229.127323] LNet: Removed LNI 192.168.203.105@tcp [ 7229.980279] Key type .llcrypt unregistered [ 7229.985033] Key type ._llcrypt unregistered [ 7255.169814] Key type ._llcrypt registered [ 7255.174568] Key type .llcrypt registered [ 7255.301164] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 7275.944875] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 7277.534397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7277.555429] alg: No test for adler32 (adler32-zlib) [ 7278.579548] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 7278.881658] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 7280.679173] Key type lgssc registered [ 7281.813769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7316.962935] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7095 sec ========== 18:29:19 (1786832959) [ 7319.133713] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:29:21 (1786832961) === [ 7322.587926] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:29:25 (1786832965) === [ 7366.062208] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 7368.910165] Key type lgssc unregistered [ 7369.233247] LNet: 243021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7369.243889] LNetError: 243021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7369.255688] LNet: Removed LNI 192.168.203.105@tcp [ 7370.093195] Key type .llcrypt unregistered [ 7370.099692] Key type ._llcrypt unregistered