[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 541828862 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003192] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.008495] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009031] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011164] LSM: Security Framework initializing [ 0.012061] Yama: becoming mindful. [ 0.014037] SELinux: Initializing. [ 0.015091] *** VALIDATE selinux *** [ 0.024798] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031102] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036075] *** VALIDATE tmpfs *** [ 0.037521] *** VALIDATE proc *** [ 0.038301] *** VALIDATE cgroup *** [ 0.039012] *** VALIDATE cgroup2 *** [ 0.040333] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043030] Spectre V2 : User space: Vulnerable [ 0.044014] Speculative Store Bypass: Vulnerable [ 0.047076] debug: unmapping init [mem 0xffffffffb8e59000-0xffffffffb8e60fff] [ 0.050216] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051829] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052029] ... version: 2 [ 0.053016] ... bit width: 48 [ 0.054011] ... generic registers: 4 [ 0.055018] ... value mask: 0000ffffffffffff [ 0.056019] ... max period: 00007fffffffffff [ 0.057018] ... fixed-purpose events: 3 [ 0.058021] ... event mask: 000000070000000f [ 0.059321] rcu: Hierarchical SRCU implementation. [ 0.061603] smp: Bringing up secondary CPUs ... [ 0.062601] x86: Booting SMP configuration: [ 0.063028] .... node #0, CPUs: #1 #2 #3 [ 0.073106] smp: Brought up 1 node, 4 CPUs [ 0.075018] smpboot: Max logical packages: 1 [ 0.076019] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.121024] node 0 deferred pages initialised in 43ms [ 0.125277] devtmpfs: initialized [ 0.126251] x86/mm: Memory block size: 128MB [ 0.130427] gcov: version magic: 0x41383552 [ 0.133370] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.134182] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.135562] pinctrl core: initialized pinctrl subsystem [ 0.136340] [ 0.136889] ************************************************************* [ 0.137013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138021] ** ** [ 0.139017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.140024] ** ** [ 0.141019] ** This means that this kernel is built to expose internal ** [ 0.142016] ** IOMMU data structures, which may compromise security on ** [ 0.143021] ** your system. ** [ 0.144020] ** ** [ 0.145022] ** If you see this message and you are not debugging the ** [ 0.146015] ** kernel, report this immediately to your vendor! ** [ 0.147037] ** ** [ 0.148028] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149015] ************************************************************* [ 0.151455] NET: Registered protocol family 16 [ 0.154000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.155080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.156085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.158328] cpuidle: using governor menu [ 0.168045] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.172877] PCI: Using configuration type 1 for base access [ 0.177153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.197084] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.199031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.206035] cryptd: max_cpu_qlen set to 1000 [ 0.208246] ACPI: Added _OSI(Module Device) [ 0.209020] ACPI: Added _OSI(Processor Device) [ 0.210016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.211016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.216399] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.222474] ACPI: Interpreter enabled [ 0.223086] ACPI: PM: (supports S0 S3 S4 S5) [ 0.224018] ACPI: Using IOAPIC for interrupt routing [ 0.225106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.226457] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.246372] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.248030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.249033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.250087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.254099] acpiphp: Slot [2] registered [ 0.256081] acpiphp: Slot [5] registered [ 0.257200] acpiphp: Slot [6] registered [ 0.258376] acpiphp: Slot [7] registered [ 0.259341] acpiphp: Slot [8] registered [ 0.260383] acpiphp: Slot [9] registered [ 0.261189] acpiphp: Slot [10] registered [ 0.262139] acpiphp: Slot [3] registered [ 0.263159] acpiphp: Slot [4] registered [ 0.264138] acpiphp: Slot [11] registered [ 0.265174] acpiphp: Slot [12] registered [ 0.266190] acpiphp: Slot [13] registered [ 0.267138] acpiphp: Slot [14] registered [ 0.268080] acpiphp: Slot [15] registered [ 0.270000] acpiphp: Slot [16] registered [ 0.273161] acpiphp: Slot [17] registered [ 0.278216] acpiphp: Slot [18] registered [ 0.282284] acpiphp: Slot [19] registered [ 0.284171] acpiphp: Slot [20] registered [ 0.288528] acpiphp: Slot [21] registered [ 0.294211] acpiphp: Slot [22] registered [ 0.299176] acpiphp: Slot [23] registered [ 0.304150] acpiphp: Slot [24] registered [ 0.315128] acpiphp: Slot [25] registered [ 0.316148] acpiphp: Slot [26] registered [ 0.317000] acpiphp: Slot [27] registered [ 0.328368] acpiphp: Slot [28] registered [ 0.333196] acpiphp: Slot [29] registered [ 0.342160] acpiphp: Slot [30] registered [ 0.343308] acpiphp: Slot [31] registered [ 0.344130] PCI host bridge to bus 0000:00 [ 0.345000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.350378] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.356028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.368031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.380027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.388037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.392221] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.393000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.410842] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.442678] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.461068] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.463031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.469022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.479020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.488099] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.498285] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.508132] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.511109] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 12695 usecs [ 0.523556] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.532016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.571016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.582027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.594077] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.623035] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.645017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.718013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.757342] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.776025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.791036] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.855029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.878000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.895113] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.921024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.007030] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 1.030273] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.044068] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.051022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.081021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.093000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.103020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.111030] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.156133] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.179575] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.198026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.213033] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.284033] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.345283] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.347456] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.350659] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.352791] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.354333] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.361039] iommu: Default domain type: Passthrough [ 1.366077] SCSI subsystem initialized [ 1.368713] ACPI: bus type USB registered [ 1.371272] usbcore: registered new interface driver usbfs [ 1.379161] usbcore: registered new interface driver hub [ 1.384184] usbcore: registered new device driver usb [ 1.388495] pps_core: LinuxPPS API ver. 1 registered [ 1.393023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.408135] PTP clock support registered [ 1.410151] EDAC MC: Ver: 3.0.0 [ 1.412208] PCI: Using ACPI for IRQ routing [ 1.414735] NetLabel: Initializing [ 1.415014] NetLabel: domain hash size = 128 [ 1.416025] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.417210] NetLabel: unlabeled traffic allowed by default [ 1.421110] vgaarb: loaded [ 1.428432] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.431020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.447421] clocksource: Switched to clocksource kvm-clock [ 1.895845] VFS: Disk quotas dquot_6.6.0 [ 1.902205] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.912611] *** VALIDATE ramfs *** [ 1.918836] *** VALIDATE hugetlbfs *** [ 1.923783] pnp: PnP ACPI init [ 1.930715] pnp: PnP ACPI: found 6 devices [ 2.020200] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 2.029584] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 2.032675] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 2.036739] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 2.046208] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 2.053915] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 2.058717] NET: Registered protocol family 2 [ 2.072874] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 2.097817] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 2.116334] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 2.125023] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 2.130269] TCP: Hash tables configured (established 65536 bind 65536) [ 2.133783] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 2.138090] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.142404] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.144934] NET: Registered protocol family 1 [ 2.153525] RPC: Registered named UNIX socket transport module. [ 2.157516] RPC: Registered udp transport module. [ 2.159517] RPC: Registered tcp transport module. [ 2.162982] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.165942] NET: Registered protocol family 44 [ 2.167972] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 2.170888] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 2.173655] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 2.176508] PCI: CLS 0 bytes, default 64 [ 2.178624] Unpacking initramfs... [ 5.940195] debug: unmapping init [mem 0xffff8bbcfcc54000-0xffff8bbcfffbffff] [ 5.971208] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.973818] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.981832] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 6.666123] hrtimer: interrupt took 3464153 ns [ 8.370126] Initialise system trusted keyrings [ 8.376132] Key type blacklist registered [ 8.382555] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 8.401911] zbud: loaded [ 8.409176] *** VALIDATE nfs *** [ 8.413187] *** VALIDATE nfs4 *** [ 8.418061] pstore: using deflate compression [ 8.430588] Platform Keyring initialized [ 8.748492] NET: Registered protocol family 38 [ 8.753982] Key type asymmetric registered [ 8.757752] Asymmetric key parser 'x509' registered [ 8.761241] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 8.767803] io scheduler mq-deadline registered [ 8.771154] io scheduler kyber registered [ 8.774995] io scheduler bfq registered [ 8.778407] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 8.789707] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 8.797598] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 8.808761] ACPI: Power Button [PWRF] [ 8.823093] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 8.835295] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 8.862538] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 8.876721] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 8.914721] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.945018] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 9.046131] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 9.070448] Non-volatile memory driver v1.3 [ 9.086926] Linux agpgart interface v0.103 [ 9.192546] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 9.198542] vda: detected capacity change from 0 to 76775424 [ 9.288549] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 9.299752] vdb: detected capacity change from 0 to 1073741824 [ 9.397262] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.408210] vdc: detected capacity change from 0 to 2621440000 [ 9.476251] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.482835] vdd: detected capacity change from 0 to 2621440000 [ 9.519106] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.524034] vde: detected capacity change from 0 to 4294967296 [ 9.590055] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.601818] vdf: detected capacity change from 0 to 4294967296 [ 9.639217] libphy: Fixed MDIO Bus: probed [ 9.697664] usbcore: registered new interface driver usbserial_generic [ 9.701073] usbserial: USB Serial support registered for generic [ 9.704908] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 9.708976] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.710498] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 9.713695] mousedev: PS/2 mouse device common for all mice [ 9.716837] rtc_cmos 00:05: RTC can wake from S4 [ 9.721578] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 9.721598] rtc_cmos 00:05: registered as rtc0 [ 9.740308] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 9.751275] intel_pstate: CPU model not supported [ 9.751950] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 9.783421] hid: raw HID events driver (C) Jiri Kosina [ 9.786716] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 9.794185] usbcore: registered new interface driver usbhid [ 9.814344] usbhid: USB HID core driver [ 9.826431] drop_monitor: Initializing network drop monitor service [ 9.845311] Initializing XFRM netlink socket [ 9.852149] NET: Registered protocol family 10 [ 9.866455] Segment Routing with IPv6 [ 9.872596] NET: Registered protocol family 17 [ 9.888876] mpls_gso: MPLS GSO support [ 9.931260] RAS: Correctable Errors collector initialized. [ 9.933203] AVX version of gcm_enc/dec engaged. [ 9.944913] AES CTR mode by8 optimization enabled [ 10.143389] sched_clock: Marking stable (10143368121, 0)->(12203286109, -2059917988) [ 10.152207] registered taskstats version 1 [ 10.154823] Loading compiled-in X.509 certificates [ 10.157717] zswap: loaded using pool lzo/zbud [ 10.228165] Key type big_key registered [ 10.264684] Key type encrypted registered [ 10.266735] ima: No TPM chip found, activating TPM-bypass! [ 10.271776] ima: Allocated hash algorithm: sha1 [ 10.274846] ima: No architecture policies found [ 10.277436] evm: Initialising EVM extended attributes: [ 10.279985] evm: security.selinux [ 10.281719] evm: security.ima [ 10.283482] evm: security.capability [ 10.284968] evm: HMAC attrs: 0x1 [ 10.295673] rtc_cmos 00:05: setting system clock to 2026-09-10 18:08:53 UTC (1789063733) [ 10.305586] debug: unmapping init [mem 0xffffffffb9e03000-0xffffffffb9ffffff] [ 10.309341] debug: unmapping init [mem 0xffffffffb8b82000-0xffffffffb8e58fff] [ 10.317142] Write protecting the kernel read-only data: 28672k [ 10.326712] debug: unmapping init [mem 0xffffffffb7203000-0xffffffffb73fffff] [ 10.334088] debug: unmapping init [mem 0xffffffffb7b14000-0xffffffffb7bfffff] [ 10.440238] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.475672] systemd[1]: Detected virtualization kvm. [ 10.486055] systemd[1]: Detected architecture x86-64. [ 10.490509] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 10.544973] systemd[1]: No hostname configured. [ 10.547724] systemd[1]: Set hostname to . [ 10.552974] random: systemd: uninitialized urandom read (16 bytes read) [ 10.560941] systemd[1]: Initializing machine ID from random generator. [ 10.748538] random: ln: uninitialized urandom read (6 bytes read) [ 10.996251] random: systemd: uninitialized urandom read (16 bytes read) [ 11.003422] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 11.014617] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 11.029573] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 12.635258] device-mapper: uevent: version 1.0.3 [ 12.637596] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 13.932184] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 14.103691] random: fast init done [ 14.203587] scsi host0: ata_piix [ 14.401886] scsi host1: ata_piix [ 14.410115] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 14.425079] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 19.807494] random: crng init done [ 19.815980] random: 7 urandom warning(s) missed due to ratelimiting [ 20.020091] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 21.944255] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.643967] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.836671] SELinux: Disabled at runtime. [ 26.063423] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.088574] systemd[1]: Detected virtualization kvm. [ 26.099176] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.976910] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.985469] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.008419] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.021691] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.029912] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.063023] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.079658] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ 28.643561] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 29.504775] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.655027] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.692348] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.102992] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.199353] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 36.736393] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.639543] NFS: Registering the id_resolver key type [ 37.642667] Key type id_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.648210] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg146-server login: [ 103.796499] libcfs: loading out-of-tree module taints kernel. [ 103.873768] Key type ._llcrypt registered [ 103.876374] Key type .llcrypt registered [ 104.001338] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 127.225833] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 128.934849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 128.970641] alg: No test for adler32 (adler32-zlib) [ 130.687765] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 131.683032] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 133.568462] Key type lgssc registered [ 135.514314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 158.207117] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 206.365268] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 222.295987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 222.380917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.793984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 223.830104] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 223.999311] Lustre: lustre-MDT0000: new disk, initializing [ 224.106855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.150869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 229.113750] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.764410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 242.918084] Lustre: 6530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 242.976259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 242.982922] Lustre: Skipped 1 previous similar message [ 243.083491] Lustre: lustre-MDT0001: new disk, initializing [ 243.144390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.217258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 243.223945] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 248.724766] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.310969] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 268.220490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 268.881976] Lustre: lustre-OST0000: new disk, initializing [ 268.891259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 268.898263] Lustre: 8468:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 269.017600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 276.065848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 276.085702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 276.182700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 278.566547] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 295.876036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 296.013818] Lustre: lustre-OST0001: new disk, initializing [ 296.017288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 296.022599] Lustre: 9542:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 296.100908] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 303.182224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 303.192629] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 303.239190] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 304.420675] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 318.069359] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 326.787466] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 336.020895] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing check_logdir /tmp/testlogs/ [ 345.260211] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing yml_node [ 354.073579] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 358.537468] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 362.382852] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 364.340430] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 14:14:46 EDT 2026 [ 383.768488] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 385.472248] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 401.889117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 401.895790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.910150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 405.477206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 405.477388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 405.487069] Lustre: Skipped 1 previous similar message [ 405.495599] Lustre: Skipped 1 previous similar message [ 407.654567] Lustre: server umount lustre-MDT0000 complete [ 411.524476] LustreError: 6523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064134 with bad export cookie 17893354524422202574 [ 411.531521] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 411.538144] LustreError: 6523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 415.713237] LustreError: 8478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 415.715829] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 415.723556] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 415.723566] Lustre: Skipped 2 previous similar messages [ 415.731877] LustreError: 8478:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 415.761696] Lustre: Skipped 2 previous similar messages [ 416.745707] LustreError: 6538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 416.768629] LustreError: 6538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 418.234041] Lustre: server umount lustre-MDT0001 complete [ 423.459613] Lustre: server umount lustre-OST0000 complete [ 429.006744] Lustre: server umount lustre-OST0001 complete [ 438.026175] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 454.738904] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 495.138853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.454368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 495.486325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 495.600504] Lustre: lustre-MDT0000: new disk, initializing [ 495.706653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.739692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 500.709246] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 512.336346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.470801] Lustre: 15121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 512.541587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 512.547110] Lustre: Skipped 1 previous similar message [ 512.681880] Lustre: lustre-MDT0001: new disk, initializing [ 512.839185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 512.885686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 512.897673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 517.841525] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 528.371852] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 537.144219] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 545.938369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 546.242551] Lustre: lustre-OST0000: new disk, initializing [ 546.244634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 546.247443] Lustre: 17089:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 546.394979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.939388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 547.963989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 548.099041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 553.751319] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 565.459136] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 571.111698] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 571.407743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 576.284529] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 576.462240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 579.045317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 579.047901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.058466] LustreError: Skipped 1 previous similar message [ 579.071059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 579.074666] Lustre: Skipped 1 previous similar message [ 579.096027] Lustre: Skipped 4 previous similar messages [ 584.168188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 584.179649] Lustre: Skipped 1 previous similar message [ 584.878903] Lustre: server umount lustre-OST0000 complete [ 589.284606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.409407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 594.419390] Lustre: Skipped 3 previous similar messages [ 595.313721] Lustre: server umount lustre-MDT0000 complete [ 599.523981] LustreError: 15131:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.546569] LustreError: 15131:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 599.803934] LustreError: 15113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064323 with bad export cookie 17893354524422204100 [ 599.820595] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.827164] LustreError: 15113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 600.249472] Lustre: server umount lustre-MDT0001 complete [ 604.573065] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 14:18:46 (1789064326) [ 611.214035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.615930] LustreError: 19357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.675556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 616.944187] LustreError: 19358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.015629] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 626.147752] LustreError: 19358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.173391] LustreError: 19358:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 626.648596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 631.790858] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 640.882444] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 646.708577] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 654.682220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 661.963275] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 670.184103] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 681.955860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 681.964790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 681.977934] Lustre: Skipped 1 previous similar message [ 681.986199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 687.095513] Lustre: server umount lustre-OST0000 complete [ 697.335872] Lustre: server umount lustre-MDT0000 complete [ 698.338794] LustreError: 19358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 698.367805] LustreError: 19358:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 701.148590] LustreError: 22140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064424 with bad export cookie 17893354524422204989 [ 701.159763] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 701.169973] LustreError: 22140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 701.618603] Lustre: server umount lustre-MDT0001 complete [ 714.692594] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 717.191905] Key type lgssc unregistered [ 717.450141] LNet: 22892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.454493] LNetError: 22892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 717.471478] LNet: Removed LNI 192.168.201.146@tcp [ 718.188141] Key type .llcrypt unregistered [ 718.189861] Key type ._llcrypt unregistered [ 745.996793] Key type ._llcrypt registered [ 745.998727] Key type .llcrypt registered [ 746.079542] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 747.728181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 747.766604] alg: No test for adler32 (adler32-zlib) [ 748.843441] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 749.097702] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 750.816236] Key type lgssc registered [ 752.067187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 759.984592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 760.002514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 761.645487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 766.779415] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 771.051453] LustreError: 24307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 771.100226] LustreError: 24307:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 776.161875] LustreError: 24308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 776.322045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 776.647104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 780.936885] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 789.347763] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 796.198388] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 803.292000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 803.786526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 806.855758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 810.148272] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 819.049501] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 837.604270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 837.607771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 837.637631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 839.040280] Lustre: server umount lustre-OST0000 complete [ 846.820548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 846.832754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 846.856840] Lustre: Skipped 1 previous similar message [ 846.868128] Lustre: Skipped 1 previous similar message [ 851.939971] LustreError: 25328:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 851.953642] LustreError: 25328:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 852.000517] Lustre: server umount lustre-MDT0000 complete [ 855.952536] LustreError: 24289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064579 with bad export cookie 7010242063181265487 [ 855.955351] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 855.963955] LustreError: 24289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 866.692692] Lustre: server umount lustre-MDT0001 complete [ 880.238181] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 882.981664] Key type lgssc unregistered [ 883.315512] LNet: 28284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.325450] LNetError: 28284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.350533] LNet: Removed LNI 192.168.201.146@tcp [ 884.140343] Key type .llcrypt unregistered [ 884.143973] Key type ._llcrypt unregistered [ 915.009376] Key type ._llcrypt registered [ 915.010994] Key type .llcrypt registered [ 915.137561] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 916.641447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 916.740667] alg: No test for adler32 (adler32-zlib) [ 917.823469] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 918.081563] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 919.752299] Key type lgssc registered [ 921.315702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 932.064944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 932.094649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 933.661441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 940.087666] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 943.077167] LustreError: 29698:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 943.084733] LustreError: 29698:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 948.209063] LustreError: 29699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.637930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.418782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 958.398076] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 968.910276] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 975.873367] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 993.731719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 994.150274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1001.627567] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1003.407407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1011.482830] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1022.948249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1022.961911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1022.975516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1023.987036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.991588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1024.008533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1024.017667] Lustre: Skipped 1 previous similar message [ 1025.729460] Lustre: server umount lustre-OST0000 complete [ 1034.212049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1034.212498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1034.218281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1034.227673] Lustre: Skipped 1 previous similar message [ 1036.675853] Lustre: server umount lustre-MDT0000 complete [ 1039.332287] LustreError: 29698:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.357253] LustreError: 29698:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1041.212690] LustreError: 29678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064764 with bad export cookie 871863250056077621 [ 1041.221793] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1041.229725] LustreError: 29678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1041.755825] Lustre: server umount lustre-MDT0001 complete [ 1055.892477] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1059.018162] Key type lgssc unregistered [ 1059.452575] LNet: 33580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.462655] LNetError: 33580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.496355] LNet: Removed LNI 192.168.201.146@tcp [ 1060.968761] Key type .llcrypt unregistered [ 1060.973950] Key type ._llcrypt unregistered [ 1094.820576] Key type ._llcrypt registered [ 1094.823551] Key type .llcrypt registered [ 1094.955479] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1096.296039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1096.458786] alg: No test for adler32 (adler32-zlib) [ 1097.570966] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 1097.796979] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1099.576238] Key type lgssc registered [ 1101.117161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1109.575355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1109.601570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.215093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.914145] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1121.770630] LustreError: 34989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1121.799086] LustreError: 34989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1126.142821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.551550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1131.892347] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1140.933334] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1148.452831] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1155.395985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1155.897326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.926466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1162.191346] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1171.664037] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1182.692559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1182.701167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.716159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1184.545794] Lustre: server umount lustre-OST0000 complete [ 1191.907029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.909434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1191.941567] Lustre: Skipped 1 previous similar message [ 1191.962334] Lustre: Skipped 1 previous similar message [ 1197.032898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1197.040472] Lustre: Skipped 1 previous similar message [ 1202.145628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1202.150661] Lustre: 34302:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789064909/real 1789064909] req@000000004f61306d x1875970432725888/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789064925 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1202.152634] Lustre: Skipped 1 previous similar message [ 1202.178275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.168935] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1203.463963] Lustre: server umount lustre-MDT0000 complete [ 1207.271704] LustreError: 34985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.300547] LustreError: 34985:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1208.290256] Lustre: 34300:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789064915/real 1789064915] req@00000000e32f2920 x1875970432728192/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789064931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1208.296145] LustreError: 34970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064931 with bad export cookie 3387907315803264483 [ 1208.343454] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1208.821948] Lustre: server umount lustre-MDT0001 complete [ 1222.200208] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1225.314832] Key type lgssc unregistered [ 1225.702677] LNet: 38523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1225.711153] LNetError: 38523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1225.731531] LNet: Removed LNI 192.168.201.146@tcp [ 1226.746771] Key type .llcrypt unregistered [ 1226.748304] Key type ._llcrypt unregistered [ 1258.493124] Key type ._llcrypt registered [ 1258.495208] Key type .llcrypt registered [ 1258.550981] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1260.092087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1260.144456] alg: No test for adler32 (adler32-zlib) [ 1261.273789] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 1261.543966] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1263.232773] Key type lgssc registered [ 1264.977793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1272.855603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1272.897639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.463845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.655846] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1284.582738] LustreError: 39934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.603159] LustreError: 39934:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1289.697661] LustreError: 39935:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.170686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.591288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1296.440978] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1305.367853] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1312.897178] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1320.421285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1320.751502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1327.836100] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1336.605671] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1344.324942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1355.745822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1355.755430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.783838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1357.146440] Lustre: server umount lustre-OST0000 complete [ 1376.224162] Lustre: 39246:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789065083/real 1789065083] req@000000005eda8c7e x1875970604694912/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789065099 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1376.249291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.322931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.357471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1384.809601] Lustre: server umount lustre-MDT0000 complete [ 1385.445975] LustreError: 39934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1385.473223] LustreError: 39934:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1388.986140] LustreError: 39915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065112 with bad export cookie 14653635262551174437 [ 1388.995505] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1403.360374] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1403.885894] Lustre: server umount lustre-MDT0001 complete [ 1417.795432] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1420.742688] Key type lgssc unregistered [ 1421.185089] LNet: 43665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1421.191382] LNetError: 43665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1421.208722] LNet: Removed LNI 192.168.201.146@tcp [ 1422.222189] Key type .llcrypt unregistered [ 1422.225187] Key type ._llcrypt unregistered [ 1449.865118] Key type ._llcrypt registered [ 1449.871185] Key type .llcrypt registered [ 1450.052860] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1451.471399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1451.516750] alg: No test for adler32 (adler32-zlib) [ 1452.783360] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 1453.153575] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1454.960283] Key type lgssc registered [ 1456.927797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1467.136681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1467.180789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1468.903599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1475.743541] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1479.139371] LustreError: 45076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.185798] LustreError: 45076:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1484.266484] LustreError: 45077:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1486.444346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1486.824511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1491.995318] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1500.705389] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1509.232759] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1516.721291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1517.011641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1523.473321] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1531.510439] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1539.581838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1548.267660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1548.275383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1548.287587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1549.798660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1549.804203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1549.817865] Lustre: Skipped 1 previous similar message [ 1549.840052] Lustre: Skipped 2 previous similar messages [ 1552.635443] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.46@tcp (stopping) [ 1554.660592] Lustre: server umount lustre-MDT0000 complete [ 1554.919893] LustreError: 45076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1554.939690] LustreError: 45076:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1557.704982] LustreError: 45072:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.186111] LustreError: 45055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065281 with bad export cookie 7651362649234176604 [ 1558.217882] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1560.033180] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.069276] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1562.832528] LustreError: 45073:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1562.861761] LustreError: 45073:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1564.622609] Lustre: server umount lustre-MDT0001 complete [ 1586.958460] Lustre: server umount lustre-OST0000 complete [ 1607.964464] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1611.235892] Key type lgssc unregistered [ 1611.628691] LNet: 48962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.638649] LNetError: 48962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1611.652178] LNet: Removed LNI 192.168.201.146@tcp [ 1612.566294] Key type .llcrypt unregistered [ 1612.567991] Key type ._llcrypt unregistered [ 1641.927345] Key type ._llcrypt registered [ 1641.929487] Key type .llcrypt registered [ 1642.018811] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1643.619396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1643.642759] alg: No test for adler32 (adler32-zlib) [ 1644.842750] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 1645.047797] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1646.734064] Key type lgssc registered [ 1648.134825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1656.615160] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1656.646618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1672.289589] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1677.408425] LustreError: 50346:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.146@tcp: failed processing log, type 4: rc = -110 [ 1704.035158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1711.204645] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1720.338931] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1889.381818] Lustre: server umount lustre-OST0000 complete [ 1913.003158] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1916.161766] Key type lgssc unregistered [ 1916.515681] LNet: 52027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.519931] LNetError: 52027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.539827] LNet: Removed LNI 192.168.201.146@tcp [ 1917.275797] Key type .llcrypt unregistered [ 1917.278632] Key type ._llcrypt unregistered [ 1943.742191] Key type ._llcrypt registered [ 1943.744162] Key type .llcrypt registered [ 1943.854614] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1945.020713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1945.066727] alg: No test for adler32 (adler32-zlib) [ 1946.269702] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 1946.525252] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1948.208319] Key type lgssc registered [ 1949.464311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1957.308920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1957.339292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.765777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.635924] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1968.100498] LustreError: 53435:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1968.129195] LustreError: 53435:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1973.227315] LustreError: 53436:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1973.264432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.795808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1978.450189] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1987.005402] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1992.613993] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1999.433858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1999.708151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.840264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2007.340816] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2016.559828] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2019.297928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2019.302285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.325578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2023.396278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2023.398523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2023.405884] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.428534] Lustre: Skipped 1 previous similar message [ 2025.590635] Lustre: server umount lustre-OST0000 complete [ 2030.051734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2030.058439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2030.067539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2033.633809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2033.636568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.716801] Lustre: server umount lustre-MDT0000 complete [ 2038.753782] LustreError: 54438:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2038.768014] LustreError: 54438:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2039.886176] LustreError: 53416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065763 with bad export cookie 14648011053199584349 [ 2039.893810] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2039.905992] LustreError: 53416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2040.405609] Lustre: server umount lustre-MDT0001 complete [ 2054.183915] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 2057.315509] Key type lgssc unregistered [ 2057.720370] LNet: 56967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2057.729841] LNetError: 56967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2057.751688] LNet: Removed LNI 192.168.201.146@tcp [ 2058.672311] Key type .llcrypt unregistered [ 2058.674307] Key type ._llcrypt unregistered [ 2090.517801] Key type ._llcrypt registered [ 2090.522068] Key type .llcrypt registered [ 2090.681205] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 2091.938898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2091.984879] alg: No test for adler32 (adler32-zlib) [ 2093.109467] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 2093.481630] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 2095.256201] Key type lgssc registered [ 2096.744801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2105.581332] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2105.622948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2121.312666] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2126.433130] LustreError: 58354:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.146@tcp: failed processing log, type 4: rc = -110 [ 2153.120477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2159.182124] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2165.918289] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2173.325251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.463744] LustreError: 57688:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000986653ce x1875971477080320/t0(0) o250->MGC192.168.201.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2173.762371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2173.823795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2177.882283] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2179.041516] LustreError: 58379:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.066680] LustreError: 58379:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2184.163862] LustreError: 58378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.263727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.790513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2193.153347] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2202.194716] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2209.348129] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2213.348294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2213.359767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.385598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2214.370236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2214.371857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2214.390770] Lustre: Skipped 1 previous similar message [ 2219.502224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2219.506705] Lustre: Skipped 1 previous similar message [ 2224.617346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2226.144281] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2226.339708] Lustre: server umount lustre-OST0000 complete [ 2254.305170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2254.324660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2254.351787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2254.358551] Lustre: Skipped 1 previous similar message [ 2257.047532] Lustre: server umount lustre-MDT0000 complete [ 2261.144477] LustreError: 58361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065984 with bad export cookie 12598818182116209515 [ 2261.148297] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2261.157472] LustreError: 58361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2267.742778] Lustre: server umount lustre-MDT0001 complete [ 2279.547510] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 2282.493641] Key type lgssc unregistered [ 2282.850223] LNet: 62097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2282.860552] LNetError: 62097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2282.875348] LNet: Removed LNI 192.168.201.146@tcp [ 2283.681085] Key type .llcrypt unregistered [ 2283.682950] Key type ._llcrypt unregistered [ 2308.511235] Key type ._llcrypt registered [ 2308.516492] Key type .llcrypt registered [ 2308.716871] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 2310.088981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2310.107479] alg: No test for adler32 (adler32-zlib) [ 2311.774389] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 2312.335247] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 2314.193589] Key type lgssc registered [ 2315.922947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2326.935537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2326.971739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.698507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2333.812197] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2338.273975] LustreError: 63524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.305836] LustreError: 63524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2343.405244] LustreError: 63523:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.461924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.806620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2351.233725] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2360.857481] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2367.535729] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2375.439302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2375.931336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.616585] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2394.254595] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2398.526822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2414.076325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2414.082652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.117270] LustreError: 65346:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.334656] Lustre: server umount lustre-OST0000 complete [ 2419.171059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2419.174565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.186372] Lustre: Skipped 2 previous similar messages [ 2419.197649] Lustre: Skipped 1 previous similar message [ 2424.290584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2424.302035] Lustre: Skipped 1 previous similar message [ 2425.347794] Lustre: server umount lustre-MDT0000 complete [ 2429.303415] LustreError: 63502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066152 with bad export cookie 11578286344219985655 [ 2429.312397] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2429.321174] LustreError: 63502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2429.661367] Lustre: server umount lustre-MDT0001 complete [ 2442.328882] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 2445.182577] Key type lgssc unregistered [ 2445.512293] LNet: 67080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2445.522343] LNetError: 67080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2445.540913] LNet: Removed LNI 192.168.201.146@tcp [ 2446.507907] Key type .llcrypt unregistered [ 2446.510586] Key type ._llcrypt unregistered [ 2481.988496] Key type ._llcrypt registered [ 2481.994447] Key type .llcrypt registered [ 2482.146821] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 2483.612335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2483.876807] alg: No test for adler32 (adler32-zlib) [ 2485.089728] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 2485.405610] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 2487.080831] Key type lgssc registered [ 2488.016746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2498.781592] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 14:50:20 (1789066220) [ 2508.319470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2508.369542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.175787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2515.536910] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2520.551897] LustreError: 68784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.579192] LustreError: 68784:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2525.672623] LustreError: 68785:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.738167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.419598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2532.643961] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2542.608946] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2549.403734] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2556.890952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2557.245747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2563.675162] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2574.014710] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2580.812756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2590.835212] Lustre: Failing over lustre-MDT0000 [ 2591.203113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.206303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2591.220644] Lustre: Skipped 2 previous similar messages [ 2591.243889] Lustre: Skipped 2 previous similar messages [ 2591.942768] Lustre: server umount lustre-MDT0000 complete [ 2595.607876] LustreError: 70905:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.646879] LustreError: 68780:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.696227] LustreError: 68780:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2604.519303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.752649] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2604.967981] Lustre: *** cfs_fail_loc=135, val=0*** [ 2605.151612] LustreError: 71896:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2605.162842] LustreError: 71896:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.146@tcp: cfg command failed: rc = -2 [ 2605.175271] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2605.175271] [ 2605.216200] LustreError: MGC192.168.201.146@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2605.241354] LustreError: 71884:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2605.258853] LustreError: 71884:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2605.283764] LustreError: 71884:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2605.301696] LustreError: 71884:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2605.319574] LustreError: 68764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066328 with bad export cookie 12093392408180846141 [ 2605.340529] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2605.360091] Lustre: server umount lustre-MDT0000 complete [ 2605.372768] LustreError: 71884:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2605.775061] LustreError: 68781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.803460] LustreError: 68781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2614.151540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2615.264144] LustreError: 72242:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2615.291927] LustreError: 72242:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000da7753d7 x1875971888154752/t0(0) o250->MGC192.168.201.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789066338 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2615.335119] LustreError: 72242:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2615.725642] LustreError: 68781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.759908] LustreError: 68781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 2616.021528] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.46@tcp (not set up) [ 2616.226404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2616.357963] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2617.031082] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2621.432615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2621.486721] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2621.564590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2623.297564] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2636.785085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.800340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.113767] Lustre: server umount lustre-OST0000 complete [ 2647.009822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2647.011596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.018507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2647.018515] Lustre: Skipped 1 previous similar message [ 2647.064663] Lustre: Skipped 2 previous similar messages [ 2652.132093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.142258] Lustre: Skipped 2 previous similar messages [ 2652.970575] Lustre: server umount lustre-MDT0000 complete [ 2657.255086] LustreError: 68784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2657.283870] LustreError: 68784:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2658.425953] LustreError: 68766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066381 with bad export cookie 12093392408180846288 [ 2658.455175] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2658.460925] LustreError: 68766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2658.978759] Lustre: server umount lustre-MDT0001 complete [ 2673.264704] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 2677.155460] Key type lgssc unregistered [ 2677.527029] LNet: 73677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2677.549665] LNetError: 73677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2677.578125] LNet: Removed LNI 192.168.201.146@tcp [ 2678.804493] Key type .llcrypt unregistered [ 2678.806875] Key type ._llcrypt unregistered [ 2711.535230] Key type ._llcrypt registered [ 2711.540341] Key type .llcrypt registered [ 2711.710596] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 2713.078209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2713.629150] alg: No test for adler32 (adler32-zlib) [ 2714.740790] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 2714.997070] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 2716.736272] Key type lgssc registered [ 2717.805714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2727.116580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2727.155320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.701792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.639350] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2738.216924] LustreError: 75088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.248609] LustreError: 75088:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2743.280395] LustreError: 75087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.629960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.193649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2749.425968] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2760.259293] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2769.702223] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2778.383617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2778.819225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.860827] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2797.012566] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2798.276654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2803.678050] Lustre: Failing over lustre-MDT0000 [ 2803.686559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2803.698729] Lustre: Skipped 1 previous similar message [ 2803.950520] Lustre: server umount lustre-MDT0000 complete [ 2804.970376] LustreError: 75083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2808.806318] LustreError: 75082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2813.624351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2813.731444] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2813.853105] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2813.928765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2813.938978] Lustre: Skipped 2 previous similar messages [ 2813.958575] LustreError: 78177:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2813.966064] LustreError: 78177:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.146@tcp: cfg command failed: rc = -9 [ 2813.976664] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2813.976664] [ 2813.985283] LustreError: MGC192.168.201.146@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2813.997438] LustreError: 78166:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2814.008920] LustreError: 78166:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2814.023901] LustreError: 78166:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2814.029028] LustreError: 78166:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2814.048409] Lustre: server umount lustre-MDT0000 complete [ 2814.053438] LustreError: 78166:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2815.175985] LustreError: 77533:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2815.211477] LustreError: 77533:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2821.440262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2821.689972] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2821.717307] LustreError: 78524:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.201.146@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000059afa8f5) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2821.750954] LustreError: 75079:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2822.103394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2822.205833] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2825.420735] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2827.251767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2827.308329] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2827.390883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2827.597573] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2840.549424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2840.558206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.577799] Lustre: Skipped 1 previous similar message [ 2840.588422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2842.622256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2842.632537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.648863] Lustre: server umount lustre-OST0000 complete [ 2852.836735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.844688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2852.850443] Lustre: Skipped 1 previous similar message [ 2852.853335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2852.884960] Lustre: Skipped 2 previous similar messages [ 2855.428795] Lustre: server umount lustre-MDT0000 complete [ 2857.965732] LustreError: 75087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.988256] LustreError: 75087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2860.661134] LustreError: 76933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066583 with bad export cookie 15864677166000068563 [ 2860.663572] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2860.675838] LustreError: 76933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2861.122815] Lustre: server umount lustre-MDT0001 complete [ 2873.176642] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 2876.156830] Key type lgssc unregistered [ 2876.467320] LNet: 79961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2876.480241] LNetError: 79961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2876.513595] LNet: Removed LNI 192.168.201.146@tcp [ 2877.329269] Key type .llcrypt unregistered [ 2877.331595] Key type ._llcrypt unregistered [ 2892.004614] Key type ._llcrypt registered [ 2892.007607] Key type .llcrypt registered [ 2892.148695] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 2892.450423] Key type .llcrypt unregistered [ 2892.453048] Key type ._llcrypt unregistered [ 2920.182889] Key type ._llcrypt registered [ 2920.185214] Key type .llcrypt registered [ 2920.312163] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 2921.780477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2921.815987] alg: No test for adler32 (adler32-zlib) [ 2923.033231] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 2923.461088] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 2925.200321] Key type lgssc registered [ 2926.289887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2935.228165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2935.244777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.806736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2942.032341] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2946.027822] LustreError: 81865:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.067861] LustreError: 81865:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2951.159896] LustreError: 81866:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.912059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2952.639828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2958.619536] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2968.820710] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2976.120254] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2985.898922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2986.449801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2993.598750] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3003.507744] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3005.942669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 3021.287623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.290687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3021.310928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3021.316822] Lustre: Skipped 1 previous similar message [ 3025.134471] Lustre: server umount lustre-OST0000 complete [ 3029.496299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3029.509387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3029.534034] Lustre: Skipped 1 previous similar message [ 3029.539594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3031.521967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.524523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3031.588038] Lustre: Skipped 1 previous similar message [ 3035.978217] Lustre: server umount lustre-MDT0000 complete [ 3036.642929] LustreError: 81865:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.432500] LustreError: 82600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066763 with bad export cookie 1188694074623254889 [ 3040.435106] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3040.441068] LustreError: 82600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3040.836560] Lustre: server umount lustre-MDT0001 complete [ 3054.612574] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3057.913990] Key type lgssc unregistered [ 3058.330609] LNet: 85421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3058.337307] LNetError: 85421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3059.374058] LNet: Removed LNI 192.168.201.146@tcp [ 3060.264898] Key type .llcrypt unregistered [ 3060.266755] Key type ._llcrypt unregistered [ 3090.023940] Key type ._llcrypt registered [ 3090.025589] Key type .llcrypt registered [ 3090.142876] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3091.330424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3091.388729] alg: No test for adler32 (adler32-zlib) [ 3092.477609] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 3092.704389] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3094.368231] Key type lgssc registered [ 3095.806989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3103.972182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3103.999319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.543824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3111.160496] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3114.980256] LustreError: 86830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3115.011700] LustreError: 86830:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3120.103167] LustreError: 86831:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3121.729207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.140286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3126.219905] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3136.047148] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3144.412980] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3152.966500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3153.376803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3162.592964] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3172.534678] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3174.932345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3185.122843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3185.139916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3185.184566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3188.214354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3188.217133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.248217] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3190.214649] Lustre: server umount lustre-OST0000 complete [ 3198.434824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3198.462253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3198.465156] Lustre: Skipped 1 previous similar message [ 3201.161203] Lustre: server umount lustre-MDT0000 complete [ 3203.575061] LustreError: 88279:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3203.633666] LustreError: 88279:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3205.624699] LustreError: 86811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066928 with bad export cookie 17960093474079753175 [ 3205.643857] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3205.937873] Lustre: server umount lustre-MDT0001 complete [ 3221.573781] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3224.859419] Key type lgssc unregistered [ 3225.271333] LNet: 90379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3225.277690] LNetError: 90379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3226.352504] LNet: Removed LNI 192.168.201.146@tcp [ 3227.380767] Key type .llcrypt unregistered [ 3227.382662] Key type ._llcrypt unregistered [ 3255.397534] Key type ._llcrypt registered [ 3255.400048] Key type .llcrypt registered [ 3255.553540] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3256.801637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3256.876388] alg: No test for adler32 (adler32-zlib) [ 3258.164851] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 3258.512337] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3260.224406] Key type lgssc registered [ 3261.473908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3272.057266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3272.076468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3274.089884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.495010] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3284.462036] LustreError: 91789:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.484523] LustreError: 91789:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3289.580237] LustreError: 91790:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3292.285572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3292.747299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3299.274381] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3309.853760] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3317.279959] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3326.507292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3327.116063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.559507] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3343.440548] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3344.501954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3358.177855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3358.183325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3358.194513] Lustre: Skipped 1 previous similar message [ 3358.198224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3358.201035] Lustre: Skipped 1 previous similar message [ 3360.225574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3360.239927] Lustre: Skipped 1 previous similar message [ 3364.372744] Lustre: server umount lustre-OST0000 complete [ 3369.443625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3369.458675] LustreError: Skipped 1 previous similar message [ 3369.464484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.477652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3370.466442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.587442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3375.591941] Lustre: Skipped 3 previous similar messages [ 3375.721680] Lustre: server umount lustre-MDT0000 complete [ 3379.894694] LustreError: 91769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067103 with bad export cookie 1342879019794378210 [ 3379.898993] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3379.909685] LustreError: 91769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3380.340878] Lustre: server umount lustre-MDT0001 complete [ 3393.421964] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3397.766676] Key type lgssc unregistered [ 3398.190801] LNet: 95346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3398.206419] LNetError: 95346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3398.233506] LNet: Removed LNI 192.168.201.146@tcp [ 3399.695488] Key type .llcrypt unregistered [ 3399.697453] Key type ._llcrypt unregistered [ 3426.258484] Key type ._llcrypt registered [ 3426.261407] Key type .llcrypt registered [ 3426.362662] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3427.914053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3427.960493] alg: No test for adler32 (adler32-zlib) [ 3428.955356] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 3429.188758] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3430.849800] Key type lgssc registered [ 3432.669626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3444.486075] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3444.553549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3460.320561] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3465.440554] LustreError: 96731:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.146@tcp: failed processing log, type 4: rc = -110 [ 3492.128413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.212164] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3508.369659] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3514.565814] Lustre: server umount lustre-OST0000 complete [ 3524.762683] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 15:07:26 (1789067246) [ 3533.078761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3533.550726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3538.918656] LustreError: 98371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.947954] LustreError: 98371:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3539.638911] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3543.020874] LustreError: 98372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3548.132912] LustreError: 98371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3550.534109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3551.007861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3557.047277] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3568.562910] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3575.860130] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3583.554091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3583.843858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.740753] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3601.549136] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3603.353450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3618.789373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3618.791438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3618.796509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3618.815714] Lustre: Skipped 1 previous similar message [ 3623.910948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3623.916239] Lustre: Skipped 2 previous similar messages [ 3624.531289] Lustre: server umount lustre-OST0000 complete [ 3629.025203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.031527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3629.045288] Lustre: Skipped 1 previous similar message [ 3629.069644] Lustre: Skipped 1 previous similar message [ 3634.147927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3634.156705] Lustre: Skipped 1 previous similar message [ 3639.266644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3639.273256] Lustre: Skipped 1 previous similar message [ 3643.360200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3643.660579] Lustre: server umount lustre-MDT0000 complete [ 3644.385694] LustreError: 99110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3644.399968] LustreError: 99110:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3649.031126] LustreError: 98353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067372 with bad export cookie 9495403368580631268 [ 3649.050608] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3650.194850] Lustre: server umount lustre-MDT0001 complete [ 3667.933915] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3671.768762] Key type lgssc unregistered [ 3672.071179] LNet: 102082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3672.081123] LNetError: 102082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3672.113728] LNet: Removed LNI 192.168.201.146@tcp [ 3673.096314] Key type .llcrypt unregistered [ 3673.102451] Key type ._llcrypt unregistered [ 3717.609657] Key type ._llcrypt registered [ 3717.611285] Key type .llcrypt registered [ 3717.691921] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3718.339142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3718.347131] alg: No test for adler32 (adler32-zlib) [ 3719.451954] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 3719.793583] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3721.480358] Key type lgssc registered [ 3722.699985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3731.282642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3731.306622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.892685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3737.457450] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3743.209381] LustreError: 104264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3743.232616] LustreError: 104264:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3746.725475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3747.326217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3751.771647] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3760.503030] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3767.973575] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3775.778743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3776.052543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3778.093177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3782.698703] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3791.025581] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3802.600964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.608762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3802.622957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3807.421922] Lustre: server umount lustre-OST0000 complete [ 3812.835755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.852634] Lustre: Skipped 1 previous similar message [ 3812.861931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3812.872056] Lustre: Skipped 2 previous similar messages [ 3817.953878] LustreError: 104260:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3817.969681] Lustre: server umount lustre-MDT0000 complete [ 3817.975464] LustreError: 104260:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3822.079390] LustreError: 104264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3822.094525] LustreError: 104264:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3823.229276] LustreError: 104245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067546 with bad export cookie 2863984185398812729 [ 3823.246668] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3823.859817] Lustre: server umount lustre-MDT0001 complete [ 3835.526410] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3838.418910] Key type lgssc unregistered [ 3838.809723] LNet: 107794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3838.815968] LNetError: 107794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3838.832712] LNet: Removed LNI 192.168.201.146@tcp [ 3839.767352] Key type .llcrypt unregistered [ 3839.770429] Key type ._llcrypt unregistered [ 3861.656344] Key type ._llcrypt registered [ 3861.658915] Key type .llcrypt registered [ 3861.805547] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3862.983742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3863.119979] alg: No test for adler32 (adler32-zlib) [ 3864.180705] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 3864.382552] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3866.064226] Key type lgssc registered [ 3867.348910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3876.619942] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3876.659308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3892.257594] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3897.381055] LustreError: 109043:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.146@tcp: failed processing log, type 4: rc = -110 [ 3924.129305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3931.585237] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3941.232492] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3949.306713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3949.732483] LustreError: MGC192.168.201.146@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3949.742235] LustreError: 110068:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3949.747495] LustreError: 110068:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3949.753667] LustreError: 110068:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3949.826701] Lustre: server umount lustre-MDT0000 complete [ 3949.836805] LustreError: 110068:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3968.480245] Lustre: 108379:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789067675/real 1789067675] req@000000003392d0bd x1875973334109568/t0(0) o400->MGC192.168.201.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789067691 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3969.656340] Lustre: server umount lustre-OST0000 complete [ 3980.631372] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 3992.442664] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4030.120724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.303452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4030.321161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4030.365705] Lustre: lustre-MDT0000: new disk, initializing [ 4030.433141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4030.449179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4035.181504] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4045.983249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4046.160581] Lustre: 112858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4046.271390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4046.280336] Lustre: Skipped 1 previous similar message [ 4046.466884] Lustre: lustre-MDT0001: new disk, initializing [ 4046.646740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4046.738820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4046.752263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4051.802895] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4062.413585] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4071.152352] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4079.611898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4080.175599] Lustre: lustre-OST0000: new disk, initializing [ 4080.182375] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4080.188179] Lustre: 114829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4080.352197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4088.476160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4088.483565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4088.778093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4090.205532] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4100.838891] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4105.142882] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4105.416531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4109.693823] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4109.871729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4114.401321] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4114.409659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.436690] LustreError: Skipped 1 previous similar message [ 4114.447689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4114.454135] Lustre: Skipped 1 previous similar message [ 4117.756457] Lustre: server umount lustre-OST0000 complete [ 4123.621297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4123.636679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4123.647768] Lustre: Skipped 1 previous similar message [ 4123.654162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4124.642142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4124.656174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4124.663508] Lustre: Skipped 1 previous similar message [ 4127.901497] Lustre: server umount lustre-MDT0000 complete [ 4131.481386] LustreError: 113656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067854 with bad export cookie 2914791375281070506 [ 4131.481864] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4131.499336] LustreError: 113656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4131.953762] Lustre: server umount lustre-MDT0001 complete [ 4140.516479] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 15:17:43 (1789067863) [ 4144.137100] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4162.765733] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 4174.695192] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4212.349252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4212.638850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4212.673568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4212.747603] Lustre: lustre-MDT0000: new disk, initializing [ 4212.797642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4212.809678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4217.453090] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4227.845173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.984654] Lustre: 119732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4228.054519] LustreError: 119743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4228.068054] LustreError: 119743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4228.071731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4228.087520] Lustre: Skipped 1 previous similar message [ 4228.155813] Lustre: lustre-MDT0001: new disk, initializing [ 4228.243718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4228.333091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4228.349142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4233.109286] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4243.457196] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4249.793161] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4256.601817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4256.820721] Lustre: lustre-OST0000: new disk, initializing [ 4256.824616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4256.830926] Lustre: 121702:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4256.916131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4263.908644] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4266.056920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4266.073894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4266.167529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4273.385891] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4278.265437] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4278.427439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4282.676719] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4282.845678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4286.958888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4286.974121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4286.981332] Lustre: Skipped 1 previous similar message [ 4287.004805] Lustre: Skipped 1 previous similar message [ 4292.066418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4292.079411] Lustre: Skipped 1 previous similar message [ 4299.232202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4299.356622] Lustre: server umount lustre-OST0000 complete [ 4305.384335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4305.403731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4305.421815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4305.427436] Lustre: Skipped 2 previous similar messages [ 4310.046596] Lustre: server umount lustre-MDT0000 complete [ 4312.551653] LustreError: 119744:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4312.572575] LustreError: 119744:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4313.199562] LustreError: 119724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068036 with bad export cookie 2914791375281071395 [ 4313.206332] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4313.209148] LustreError: 119724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4313.546254] Lustre: server umount lustre-MDT0001 complete [ 4322.190855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4322.577302] LustreError: 123921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4322.653684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4327.910614] LustreError: 123922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4328.017695] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4332.015159] LustreError: 123921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4335.637554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4340.070242] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4347.517410] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4353.268094] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4359.667971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4365.991053] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4373.625291] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4380.226765] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4386.791563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4386.800285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4386.823626] Lustre: Skipped 1 previous similar message [ 4386.836389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4386.849267] Lustre: Skipped 2 previous similar messages [ 4389.567361] Lustre: server umount lustre-OST0000 complete [ 4397.539993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4397.546555] LustreError: Skipped 1 previous similar message [ 4399.920965] Lustre: server umount lustre-MDT0000 complete [ 4403.178256] LustreError: 124968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.041592] LustreError: 123902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068127 with bad export cookie 2914791375281072277 [ 4404.057686] LustreError: 123902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4404.062651] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4404.735958] Lustre: server umount lustre-MDT0001 complete [ 4416.468281] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 4419.720566] Key type lgssc unregistered [ 4420.002749] LNet: 127547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4420.013435] LNetError: 127547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4420.040844] LNet: Removed LNI 192.168.201.146@tcp [ 4421.357763] Key type .llcrypt unregistered [ 4421.360778] Key type ._llcrypt unregistered [ 4447.764937] Key type ._llcrypt registered [ 4447.767597] Key type .llcrypt registered [ 4447.938551] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 4466.086424] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4467.700109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4467.730653] alg: No test for adler32 (adler32-zlib) [ 4468.949825] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 4469.482223] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 4471.249123] Key type lgssc registered [ 4472.418732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4515.624576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4515.673955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4517.036414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4517.059658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4517.133991] Lustre: lustre-MDT0000: new disk, initializing [ 4517.243276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4517.274815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4522.440760] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4534.828142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4534.938558] Lustre: 130701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4534.993517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4534.997513] Lustre: Skipped 1 previous similar message [ 4535.062674] Lustre: lustre-MDT0001: new disk, initializing [ 4535.209257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4535.274448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4535.296163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4539.944654] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4549.347251] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4555.478057] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4562.936519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4563.469372] Lustre: lustre-OST0000: new disk, initializing [ 4563.484756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4563.500721] Lustre: 132666:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4563.639474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.147344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4569.169194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4569.343534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4570.572419] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4580.611216] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4585.952054] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4586.132873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4590.707508] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4590.906317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4595.179946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4595.181682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4595.210318] LustreError: Skipped 1 previous similar message [ 4595.213553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4595.222585] Lustre: Skipped 1 previous similar message [ 4598.963665] Lustre: server umount lustre-OST0000 complete [ 4605.409622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.412339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4605.423042] Lustre: Skipped 2 previous similar messages [ 4605.437561] Lustre: Skipped 1 previous similar message [ 4608.382602] Lustre: server umount lustre-MDT0000 complete [ 4611.843135] LustreError: 130691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068335 with bad export cookie 4490185188777608060 [ 4611.844675] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4611.851706] LustreError: 130691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4612.358466] Lustre: server umount lustre-MDT0001 complete [ 4621.117614] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 15:25:43 (1789068343) [ 4627.184310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.570658] LustreError: 135079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4627.581172] LustreError: 135079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4627.625795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4631.901524] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4633.059140] LustreError: 135080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4637.179888] LustreError: 135079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4640.613169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4641.170306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4646.552252] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4655.496586] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4662.828954] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4665.832795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4665.838424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4665.855845] Lustre: Skipped 1 previous similar message [ 4670.946957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4670.952612] Lustre: Skipped 2 previous similar messages [ 4676.065816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4676.073859] Lustre: Skipped 1 previous similar message [ 4679.136119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4679.498155] Lustre: server umount lustre-MDT0000 complete [ 4681.185185] LustreError: 135807:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4681.206920] LustreError: 135807:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4683.618079] LustreError: 135061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068406 with bad export cookie 4490185188777608942 [ 4683.622151] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4683.629224] LustreError: 135061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4684.153813] Lustre: server umount lustre-MDT0001 complete [ 4693.641728] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 15:26:55 (1789068415) [ 4700.476799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4716.000710] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4721.120486] LustreError: 137496:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.146@tcp: failed processing log, type 4: rc = -110 [ 4746.784319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4754.325622] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4763.132971] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4765.303598] Lustre: server umount lustre-OST0000 complete [ 4775.631482] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 15:28:17 (1789068497) [ 4783.249786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4783.563439] LustreError: 138943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4783.634795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4787.855323] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4792.809562] LustreError: 138944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4792.839282] LustreError: 138944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4796.627180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4801.288225] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4809.971602] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4816.762792] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4824.189666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4824.553568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4824.564076] Lustre: Skipped 1 previous similar message [ 4831.060860] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4838.679693] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4853.218995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4853.228902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4853.238815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4857.318167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4857.330449] Lustre: Skipped 1 previous similar message [ 4858.975531] Lustre: server umount lustre-MDT0000 complete [ 4862.436287] LustreError: 138940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4862.472514] LustreError: 138940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4863.313706] LustreError: 138925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068586 with bad export cookie 4490185188777609607 [ 4863.334946] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4863.335299] LustreError: 138925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4863.687435] Lustre: server umount lustre-MDT0001 complete [ 4878.006728] Lustre: server umount lustre-OST0000 complete [ 4887.116895] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:30:09 (1789068609) [ 4895.007751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4895.635657] LustreError: 142623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4895.701433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4900.517439] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4910.312536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4915.218268] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4924.304494] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4930.761421] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4937.776239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4940.216268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4944.946796] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4953.439765] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4958.992664] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4959.245951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4964.396316] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4964.578656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4970.986139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4970.986895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4971.004502] LustreError: Skipped 1 previous similar message [ 4971.007245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4971.021887] Lustre: Skipped 1 previous similar message [ 4971.053271] Lustre: Skipped 4 previous similar messages [ 4972.863485] Lustre: server umount lustre-OST0000 complete [ 4983.494818] Lustre: server umount lustre-MDT0000 complete [ 4984.296164] LustreError: 142623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4984.320868] LustreError: 142623:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4987.403191] LustreError: 142605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068710 with bad export cookie 4490185188777610678 [ 4987.418273] LustreError: 142605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4987.431584] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4987.659685] Lustre: server umount lustre-MDT0001 complete [ 4997.243825] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:31:59 (1789068719) [ 5004.571734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5020.192628] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5025.312483] LustreError: 146780:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.146@tcp: failed processing log, type 4: rc = -110 [ 5051.040356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5051.052704] Lustre: Skipped 2 previous similar messages [ 5057.649530] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5065.124836] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5072.489595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5073.090045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5077.692669] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5087.800746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5092.892623] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5100.252880] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5106.949775] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5112.001369] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5112.216340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5117.070463] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5117.321386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5124.065943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5124.074128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5124.081242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5124.081251] Lustre: Skipped 2 previous similar messages [ 5124.116421] Lustre: Skipped 4 previous similar messages [ 5125.733487] Lustre: server umount lustre-MDT0000 complete [ 5129.193056] LustreError: 146805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5129.216770] LustreError: 146805:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 5129.892500] LustreError: 146788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068853 with bad export cookie 4490185188777611700 [ 5129.893737] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5129.908376] LustreError: 146788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5130.277490] Lustre: server umount lustre-MDT0001 complete [ 5144.944243] Lustre: server umount lustre-OST0000 complete [ 5153.982981] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 15:34:36 (1789068876) [ 5163.060537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5178.784660] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5183.904438] LustreError: 150980:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.146@tcp: failed processing log, type 4: rc = -110 [ 5216.691915] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5225.394299] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5232.991039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5233.691552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5238.356868] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5246.795341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5251.894899] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5260.257983] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5267.644941] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5274.885577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5275.123416] Lustre: lustre-OST0001: new disk, initializing [ 5275.127506] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5275.133478] Lustre: 153852:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5280.734947] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5284.408138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5284.433636] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5284.520174] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5291.019269] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5296.347623] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5296.521301] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5300.532709] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5300.683485] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5303.266963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5303.280223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.302387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5303.309478] Lustre: Skipped 2 previous similar messages [ 5308.649269] Lustre: server umount lustre-OST0000 complete [ 5315.567508] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5315.591908] LustreError: Skipped 1 previous similar message [ 5327.328393] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5327.676202] Lustre: server umount lustre-OST0001 complete [ 5333.984883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5338.486284] Lustre: server umount lustre-MDT0000 complete [ 5341.485728] LustreError: 152762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069064 with bad export cookie 4490185188777612652 [ 5341.493793] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5341.501869] LustreError: 152762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5341.819083] Lustre: server umount lustre-MDT0001 complete [ 5367.517428] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 15:38:09 (1789069089) [ 5369.376930] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5371.159449] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5373.319772] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:38:15 (1789069095) [ 5380.445690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5380.967497] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5381.042774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5381.328652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5381.333642] Lustre: Skipped 6 previous similar messages [ 5385.831891] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5393.904330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.967188] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5393.986384] Lustre: 157411:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5398.217211] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5405.761105] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5411.535288] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5418.177301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5418.359403] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5425.409567] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5428.873465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5434.423696] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5439.939086] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5440.113548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5445.362988] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5445.669673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5449.185509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5449.195389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5449.218264] Lustre: Skipped 5 previous similar messages [ 5449.230535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5449.234075] Lustre: Skipped 11 previous similar messages [ 5453.955409] Lustre: server umount lustre-OST0000 complete [ 5476.786112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5483.568104] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5490.418033] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.441864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.449140] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5490.476333] Lustre: Skipped 1 previous similar message [ 5491.009467] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5495.962077] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5496.212362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5501.569387] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5501.805607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5507.867528] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff996c468f5800.ost_server_uuid 50 [ 5509.489497] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff996c468f5800.ost_server_uuid in FULL state after 0 sec [ 5509.862807] LustreError: 158687:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.46@tcp) returned error from blocking AST (req@00000000e44ceebf x1875973968762752 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000005509336/0x3e5057a3a0cf5ec6 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.46@tcp remote: 0x3691c05beb93cd5b expref: 5 pid: 157421 timeout: 5609 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5509.914166] LustreError: lustre-MDT0000: A client on nid 192.168.201.46@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5509.932508] LustreError: 157409:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.46@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000005509336/0x3e5057a3a0cf5ec6 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.46@tcp remote: 0x3691c05beb93cd5b expref: 6 pid: 157421 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5515.760682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5515.767415] LustreError: Skipped 1 previous similar message [ 5521.156749] Lustre: server umount lustre-OST0000 complete [ 5532.167337] Lustre: server umount lustre-MDT0000 complete [ 5536.028856] LustreError: 157405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069259 with bad export cookie 4490185188777613835 [ 5536.041914] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5536.044890] LustreError: 157405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5542.414860] Lustre: server umount lustre-MDT0001 complete [ 5554.209382] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 5556.732244] Key type lgssc unregistered [ 5557.008881] LNet: 163524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5557.017530] LNetError: 163524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5558.057439] LNet: Removed LNI 192.168.201.146@tcp [ 5559.158185] Key type .llcrypt unregistered [ 5559.163492] Key type ._llcrypt unregistered [ 5585.732762] Key type ._llcrypt registered [ 5585.739226] Key type .llcrypt registered [ 5585.828836] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5587.353634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5587.437538] alg: No test for adler32 (adler32-zlib) [ 5588.586771] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 5588.860564] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 5590.506546] Key type lgssc registered [ 5591.880573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5600.470561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5600.489985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5602.079023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5606.335396] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5612.527093] LustreError: 164936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5612.563534] LustreError: 164936:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5616.515787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5616.909815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5621.665216] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5629.840482] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5635.912937] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5643.409057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5643.725071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5646.843883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5650.434931] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5658.812573] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5668.972935] Lustre: Failing over lustre-MDT0000 [ 5669.237820] Lustre: server umount lustre-MDT0000 complete [ 5672.418249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5672.420115] LustreError: 165940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5672.427393] Lustre: Skipped 2 previous similar messages [ 5672.448899] LustreError: 165940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5677.539133] LustreError: 164935:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5677.569522] LustreError: 164935:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5681.459509] LustreError: 164916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.46@tcp arrived at 1789069404 with bad export cookie 14691443713739683345 [ 5681.499137] LustreError: 164916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5682.670871] LustreError: 164936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5682.698725] LustreError: 164936:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5687.780242] LustreError: 167428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5687.810231] LustreError: 167428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5688.803453] Lustre: 164245:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789069395/real 1789069395] req@0000000017612c10 x1875975142926720/t0(0) o400->MGC192.168.201.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789069411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5688.845985] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5689.702384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.044153] LustreError: 165940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5699.048191] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcbe27e381319767a [ 5699.067281] LustreError: 165940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 5699.084134] Lustre: MGC192.168.201.146@tcp: Connection restored to 0@lo (at 0@lo) [ 5699.465484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5699.542744] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5704.293385] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5704.681296] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5704.695044] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5707.129994] Lustre: lustre-MDT0000: Denying connection for new client 90866860-c1e5-4304-8dd9-58f61f250588 (at 192.168.201.46@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5718.498948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5718.504749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.521688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5720.034310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5720.036607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5721.933735] Lustre: server umount lustre-OST0000 complete [ 5725.868765] LustreError: 168666:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5725.878564] Lustre: 168015:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5725.886298] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5725.897486] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5725.903051] LustreError: 168015:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000086dbf155 x1875975142947584/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5725.911905] LustreError: 168015:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5725.919503] LustreError: 168015:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5725.927612] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5726.295445] Lustre: server umount lustre-MDT0000 complete [ 5730.282479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5730.320907] LustreError: 164931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5731.105966] LustreError: 164915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069454 with bad export cookie 14691443713739683450 [ 5731.114419] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5731.116515] LustreError: 164915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5731.690767] Lustre: server umount lustre-MDT0001 complete [ 5745.319074] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 5748.431495] Key type lgssc unregistered [ 5748.681690] LNet: 169417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5748.688994] LNetError: 169417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5748.708942] LNet: Removed LNI 192.168.201.146@tcp [ 5749.874648] Key type .llcrypt unregistered [ 5749.876598] Key type ._llcrypt unregistered [ 5782.380914] Key type ._llcrypt registered [ 5782.383553] Key type .llcrypt registered [ 5782.485283] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5783.453449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5783.504722] alg: No test for adler32 (adler32-zlib) [ 5784.802864] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 5785.056381] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 5786.744262] Key type lgssc registered [ 5788.137451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5796.731752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5796.760745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5798.394327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5804.255925] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5808.610854] LustreError: 170829:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5808.670222] LustreError: 170829:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5813.734019] LustreError: 170830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5815.418216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5815.774641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5821.866742] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5830.994727] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5839.016446] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5847.915534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5848.434414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5857.399442] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5865.942292] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5867.909133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5881.313768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5881.319437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5881.335837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5883.363425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5883.374158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5883.375622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.418483] Lustre: Skipped 1 previous similar message [ 5885.402939] Lustre: server umount lustre-OST0000 complete [ 5892.580311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5892.589567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5892.614738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5898.723699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5898.727240] Lustre: Skipped 3 previous similar messages [ 5903.854183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5903.862848] Lustre: Skipped 1 previous similar message [ 5904.352193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5904.597209] Lustre: server umount lustre-MDT0000 complete [ 5907.938269] LustreError: 170825:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5907.949028] LustreError: 170825:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5909.416939] LustreError: 170811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069632 with bad export cookie 17153461860416098846 [ 5909.424096] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5910.071889] Lustre: server umount lustre-MDT0001 complete [ 5925.939879] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 5929.284219] Key type lgssc unregistered [ 5929.643732] LNet: 174389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5929.649661] LNetError: 174389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5930.729815] LNet: Removed LNI 192.168.201.146@tcp [ 5931.871329] Key type .llcrypt unregistered [ 5931.873366] Key type ._llcrypt unregistered [ 5971.165993] Key type ._llcrypt registered [ 5971.169086] Key type .llcrypt registered [ 5971.307941] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5973.420246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5973.485228] alg: No test for adler32 (adler32-zlib) [ 5974.718303] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 5975.022492] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 5976.744198] Key type lgssc registered [ 5978.315367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5987.793054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5987.817023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5989.565657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5994.914275] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5999.076397] LustreError: 176102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5999.096053] LustreError: 176102:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6004.088687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6004.197377] LustreError: 176101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6004.475730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6009.105531] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6017.820305] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6024.763517] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6032.495592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6032.863202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6033.943802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 6040.641605] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6049.553721] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6059.495148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6059.500066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6059.501737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6059.514794] LustreError: Skipped 1 previous similar message [ 6059.545328] Lustre: Skipped 1 previous similar message [ 6063.756714] Lustre: server umount lustre-OST0000 complete [ 6069.730579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.767281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6069.780401] Lustre: Skipped 2 previous similar messages [ 6074.857821] LustreError: 176096:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6074.866821] LustreError: 176096:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6074.935360] Lustre: server umount lustre-MDT0000 complete [ 6078.968069] LustreError: 176097:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6078.998500] LustreError: 176097:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6079.503779] LustreError: 176082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069802 with bad export cookie 2560605336325062594 [ 6079.512942] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6079.525401] LustreError: 176082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6080.082255] Lustre: server umount lustre-MDT0001 complete [ 6093.251764] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6095.901958] Key type lgssc unregistered [ 6096.135220] LNet: 179635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6096.150736] LNetError: 179635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6096.162949] LNet: Removed LNI 192.168.201.146@tcp [ 6097.165119] Key type .llcrypt unregistered [ 6097.171824] Key type ._llcrypt unregistered [ 6122.626693] Key type ._llcrypt registered [ 6122.629227] Key type .llcrypt registered [ 6122.696171] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6123.928427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6123.976614] alg: No test for adler32 (adler32-zlib) [ 6125.059585] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 6125.248805] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 6126.904223] Key type lgssc registered [ 6128.168120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6140.627267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6140.647091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6142.122888] Lustre: *** cfs_fail_loc=135, val=0*** [ 6142.214535] LustreError: 181158:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6142.218786] LustreError: 181158:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.146@tcp: cfg command failed: rc = -2 [ 6142.223824] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6142.223824] [ 6142.229458] LustreError: MGC192.168.201.146@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6142.248535] LustreError: 181118:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6142.256589] LustreError: 181118:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 6142.275598] LustreError: 181118:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 6142.278855] LustreError: 181118:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6142.358243] Lustre: server umount lustre-MDT0000 complete [ 6142.361182] LustreError: 181118:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6149.737607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6150.123792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6155.176218] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6160.390556] LustreError: 181528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6160.428902] LustreError: 181528:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6164.773684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6165.103603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6169.639182] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6177.453562] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6184.410267] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6190.501960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6190.503470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6190.509658] Lustre: Skipped 1 previous similar message [ 6190.519711] Lustre: Skipped 1 previous similar message [ 6192.658068] Lustre: server umount lustre-MDT0000 complete [ 6195.694598] LustreError: 182766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6195.721472] LustreError: 182766:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6196.666559] LustreError: 181509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069919 with bad export cookie 17587614649639354529 [ 6196.680930] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6196.687885] LustreError: 181509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6196.976347] Lustre: server umount lustre-MDT0001 complete [ 6212.069236] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6215.386319] Key type lgssc unregistered [ 6215.813972] LNet: 183849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6215.823182] LNetError: 183849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6215.868794] LNet: Removed LNI 192.168.201.146@tcp [ 6217.053571] Key type .llcrypt unregistered [ 6217.063171] Key type ._llcrypt unregistered [ 6244.284172] Key type ._llcrypt registered [ 6244.287181] Key type .llcrypt registered [ 6244.380712] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6244.650788] Key type .llcrypt unregistered [ 6244.653233] Key type ._llcrypt unregistered [ 6265.406671] Key type ._llcrypt registered [ 6265.410331] Key type .llcrypt registered [ 6265.513920] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6267.122737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6267.391117] alg: No test for adler32 (adler32-zlib) [ 6268.648891] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 6269.016063] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 6270.784282] Key type lgssc registered [ 6272.242163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6282.393187] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6282.409253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6298.016731] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6303.136428] LustreError: 185978:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.146@tcp: failed processing log, type 4: rc = -110 [ 6329.824214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6337.805666] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6346.116320] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6354.922777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6355.665474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6355.748689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6361.059387] LustreError: 186004:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6361.089532] LustreError: 186004:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6361.553719] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6364.149301] LustreError: 186003:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6364.189377] LustreError: 186003:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6369.250233] LustreError: 186004:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6369.315913] LustreError: 186004:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6373.011266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.396687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6378.607216] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6387.954574] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6395.897894] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6399.297616] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6412.260686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6412.271115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6412.286050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6414.315169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6414.316415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6414.334718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6418.601296] Lustre: server umount lustre-OST0000 complete [ 6424.545824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6424.550109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6424.558837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6424.577380] Lustre: Skipped 1 previous similar message [ 6424.588378] Lustre: Skipped 1 previous similar message [ 6428.954748] Lustre: server umount lustre-MDT0000 complete [ 6429.671554] LustreError: 187018:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6429.692593] LustreError: 187018:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6432.845417] LustreError: 185986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070156 with bad export cookie 1794993455699928057 [ 6432.860433] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6433.340735] Lustre: server umount lustre-MDT0001 complete [ 6445.867773] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6448.366269] Key type lgssc unregistered [ 6448.598227] LNet: 189990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6448.605684] LNetError: 189990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6448.630342] LNet: Removed LNI 192.168.201.146@tcp [ 6449.319528] Key type .llcrypt unregistered [ 6449.322514] Key type ._llcrypt unregistered [ 6474.076967] Key type ._llcrypt registered [ 6474.078894] Key type .llcrypt registered [ 6474.198602] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6475.532956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6475.551075] alg: No test for adler32 (adler32-zlib) [ 6476.618523] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 6476.850229] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 6478.592215] Key type lgssc registered [ 6479.772053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6488.892913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6488.929169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6490.468753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6495.736537] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6499.811376] LustreError: 191403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6499.841095] LustreError: 191403:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6504.929510] LustreError: 191404:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6507.046224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6507.446748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6512.457945] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6521.509203] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6528.709869] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6536.944221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6537.442336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6545.546429] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6555.808524] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6560.020872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6565.978342] Lustre: Failing over lustre-MDT0000 [ 6566.645905] Lustre: server umount lustre-MDT0000 complete [ 6567.145924] LustreError: 192632:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6568.930091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6568.936378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6570.466242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6570.471159] LustreError: 191403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6570.487697] Lustre: Skipped 1 previous similar message [ 6570.535369] LustreError: 191403:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6575.587177] LustreError: 191399:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6575.615441] LustreError: 191399:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6585.825348] Lustre: 190713:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789070293/real 1789070293] req@00000000ad75db87 x1875976074064896/t0(0) o400->MGC192.168.201.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789070309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6585.832319] LustreError: 191403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6585.859212] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6585.921936] LustreError: 191403:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 6587.258087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6596.064906] LustreError: 190710:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000009ba0bbb x1875976074069632/t0(0) o250->MGC192.168.201.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6596.386794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6596.467751] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6598.863606] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6601.717465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6601.751251] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6601.834621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6602.348126] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6607.448087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6613.083559] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6619.619912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6619.625236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6619.646481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6621.467739] Lustre: server umount lustre-OST0000 complete [ 6627.297408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6627.309214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6627.316379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6627.321710] Lustre: Skipped 1 previous similar message [ 6627.340853] Lustre: Skipped 1 previous similar message [ 6632.419379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6632.431350] Lustre: Skipped 1 previous similar message [ 6635.600790] Lustre: server umount lustre-MDT0000 complete [ 6637.540586] LustreError: 191403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6637.564694] LustreError: 191403:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6640.450908] LustreError: 192141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070363 with bad export cookie 4652055126106217335 [ 6640.459901] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6640.791778] Lustre: server umount lustre-MDT0001 complete [ 6654.351332] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6658.120636] Key type lgssc unregistered [ 6658.559942] LNet: 196247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6658.574374] LNetError: 196247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6659.631738] LNet: Removed LNI 192.168.201.146@tcp [ 6660.614814] Key type .llcrypt unregistered [ 6660.617881] Key type ._llcrypt unregistered [ 6696.684347] Key type ._llcrypt registered [ 6696.686339] Key type .llcrypt registered [ 6696.856863] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6698.437565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6698.480669] alg: No test for adler32 (adler32-zlib) [ 6699.882598] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 6700.426243] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 6702.121280] Key type lgssc registered [ 6704.384635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6713.939984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6713.974395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6715.609756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6721.093834] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6725.096519] LustreError: 197665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6725.143674] LustreError: 197665:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6730.209211] LustreError: 197664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6732.023805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.506310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6738.609925] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6749.269294] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6757.974232] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6765.274338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6765.700281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6772.682458] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6780.643514] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6785.145532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6787.677403] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6791.083742] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6808.666191] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6821.344956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6821.353730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6821.367031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6821.376146] Lustre: Skipped 1 previous similar message [ 6821.398645] Lustre: Skipped 1 previous similar message [ 6826.340665] Lustre: server umount lustre-OST0000 complete [ 6831.589779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6831.615088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6831.630210] Lustre: Skipped 1 previous similar message [ 6836.706374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6836.720426] Lustre: Skipped 1 previous similar message [ 6837.111913] Lustre: server umount lustre-MDT0000 complete [ 6840.809362] LustreError: 198393:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6840.830269] LustreError: 198393:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6841.567818] LustreError: 197645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070564 with bad export cookie 4633510646825229833 [ 6841.586203] LustreError: 197645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6841.587613] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6842.199993] Lustre: server umount lustre-MDT0001 complete [ 6854.998688] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 6858.739589] Key type lgssc unregistered [ 6859.045882] LNet: 201544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6859.052889] LNetError: 201544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6859.073520] LNet: Removed LNI 192.168.201.146@tcp [ 6860.204952] Key type .llcrypt unregistered [ 6860.207219] Key type ._llcrypt unregistered [ 6883.945390] Key type ._llcrypt registered [ 6883.947162] Key type .llcrypt registered [ 6884.108668] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6885.534204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6885.776360] alg: No test for adler32 (adler32-zlib) [ 6887.001883] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 6887.321256] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 6889.049629] Key type lgssc registered [ 6890.231828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6899.499367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6899.523090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6901.286500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6906.935832] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6911.457251] LustreError: 202812:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6911.474157] LustreError: 202812:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6916.587654] LustreError: 202813:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6917.779807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.235871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6924.460602] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6933.860390] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6940.489946] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6947.727335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6948.044867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6954.779167] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6961.963427] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6971.636177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6978.394648] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6989.699170] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7002.936635] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 7013.779940] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 7025.121562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7025.142804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7025.160496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7028.194587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7028.202069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7028.206157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7030.993819] Lustre: server umount lustre-OST0000 complete [ 7035.873293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7035.880899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7035.895654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7035.903433] Lustre: Skipped 1 previous similar message [ 7038.434128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7038.435655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7038.460613] Lustre: Skipped 1 previous similar message [ 7043.554116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7043.568239] Lustre: Skipped 1 previous similar message [ 7049.184399] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7049.475357] Lustre: server umount lustre-MDT0000 complete [ 7053.198090] LustreError: 203547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070776 with bad export cookie 12883683339620378944 [ 7053.199084] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7053.205651] LustreError: 203547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7053.725889] Lustre: server umount lustre-MDT0001 complete [ 7066.729200] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7069.475729] Key type lgssc unregistered [ 7069.926137] LNet: 207902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7069.934421] LNetError: 207902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7069.948677] LNet: Removed LNI 192.168.201.146@tcp [ 7070.842701] Key type .llcrypt unregistered [ 7070.846188] Key type ._llcrypt unregistered [ 7101.539102] Key type ._llcrypt registered [ 7101.541388] Key type .llcrypt registered [ 7101.677444] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 7103.338989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7103.363490] alg: No test for adler32 (adler32-zlib) [ 7105.256044] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 7105.622594] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 7107.369214] Key type lgssc registered [ 7109.585974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7120.396909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7120.427260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7122.031036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7127.738841] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7132.661969] LustreError: 209323:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7132.682621] LustreError: 209323:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7137.622232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.769662] LustreError: 209322:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7138.200213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7143.352139] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7153.261228] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7161.315406] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7170.014071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7170.546608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7178.305606] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7188.017630] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7192.131798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7212.517239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7212.530719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7212.538954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7212.551420] Lustre: Skipped 1 previous similar message [ 7216.288714] Lustre: server umount lustre-OST0000 complete [ 7222.756251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7222.759759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7222.780681] Lustre: Skipped 1 previous similar message [ 7222.798030] Lustre: Skipped 2 previous similar messages [ 7226.909616] Lustre: server umount lustre-MDT0000 complete [ 7227.876620] LustreError: 209323:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7227.885925] LustreError: 209323:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7230.737161] LustreError: 209303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070953 with bad export cookie 5724055034938088165 [ 7230.739661] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7230.746784] LustreError: 209303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7231.109743] Lustre: server umount lustre-MDT0001 complete [ 7246.403274] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7249.574655] Key type lgssc unregistered [ 7249.923655] LNet: 213032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7249.936295] LNetError: 213032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7250.982848] LNet: Removed LNI 192.168.201.146@tcp [ 7252.189536] Key type .llcrypt unregistered [ 7252.191092] Key type ._llcrypt unregistered [ 7273.008899] Key type ._llcrypt registered [ 7273.011138] Key type .llcrypt registered [ 7273.109375] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 7274.151085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7274.311850] alg: No test for adler32 (adler32-zlib) [ 7275.415409] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 7275.653966] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 7277.320236] Key type lgssc registered [ 7279.098773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7287.747722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7287.760015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7289.339959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7294.749857] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7299.554344] LustreError: 214296:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7299.585032] LustreError: 214296:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7304.632942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7304.678794] LustreError: 214295:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7305.101226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7310.463710] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7320.139136] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7327.491159] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7336.855932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7337.412896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7346.339194] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7356.407301] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7359.972300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7380.464466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7380.481285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7380.484982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7380.504633] Lustre: Skipped 1 previous similar message [ 7384.471621] Lustre: server umount lustre-OST0000 complete [ 7390.697054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7390.707753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7390.716501] Lustre: Skipped 2 previous similar messages [ 7390.727204] Lustre: Skipped 1 previous similar message [ 7394.972518] Lustre: server umount lustre-MDT0000 complete [ 7395.810982] LustreError: 216031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7395.836086] LustreError: 216031:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7399.140673] LustreError: 214276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071122 with bad export cookie 14024786350489002449 [ 7399.149802] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7399.159626] LustreError: 214276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7399.528745] Lustre: server umount lustre-MDT0001 complete [ 7412.574802] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7415.350639] Key type lgssc unregistered [ 7415.626875] LNet: 218002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7415.629498] LNetError: 218002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7415.640138] LNet: Removed LNI 192.168.201.146@tcp [ 7416.422825] Key type .llcrypt unregistered [ 7416.426088] Key type ._llcrypt unregistered [ 7438.613304] Key type ._llcrypt registered [ 7438.615677] Key type .llcrypt registered [ 7438.727290] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 7439.795161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7439.987330] alg: No test for adler32 (adler32-zlib) [ 7441.052498] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 7441.366709] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 7443.152257] Key type lgssc registered [ 7444.175133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7452.809114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7452.844665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7454.391124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7458.326964] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7463.909363] LustreError: 219263:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7463.931246] LustreError: 219263:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7467.801586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7468.198958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7473.576960] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7483.094045] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7491.234819] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7498.857036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7499.250565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7504.380429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7505.921757] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7514.318609] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7524.259981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7524.313583] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7524.601754] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7531.364594] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7539.453586] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7541.529592] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7553.530317] Lustre: Permanently deactivating lustre-OST0001 [ 7553.545243] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7563.962049] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7572.461715] Lustre: Permanently reactivating lustre-OST0001 [ 7572.507279] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7572.525144] Lustre: Skipped 2 previous similar messages [ 7580.789074] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7580.794265] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7580.798913] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7580.807476] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7580.850610] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7589.867328] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.879475] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7589.890236] Lustre: Skipped 2 previous similar messages [ 7594.980150] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7595.010397] Lustre: Skipped 2 previous similar messages [ 7600.123921] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7600.138372] Lustre: Skipped 1 previous similar message [ 7602.148161] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7602.288093] Lustre: server umount lustre-OST0001 complete [ 7605.219385] LustreError: 221087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7605.240223] LustreError: 221087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7610.340525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7610.341123] LustreError: 223201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7610.342885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7610.390048] Lustre: Skipped 1 previous similar message [ 7610.480434] LustreError: 223201:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7614.476871] Lustre: server umount lustre-OST0000 complete [ 7620.578931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7620.582609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7620.590057] Lustre: Skipped 1 previous similar message [ 7620.592340] Lustre: Skipped 1 previous similar message [ 7625.128742] Lustre: server umount lustre-MDT0000 complete [ 7625.703596] LustreError: 223081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7625.727162] LustreError: 223081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7629.647501] LustreError: 219244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071352 with bad export cookie 4515081136946281345 [ 7629.655291] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7629.673680] LustreError: 219244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7630.051923] Lustre: server umount lustre-MDT0001 complete [ 7643.662223] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7647.288405] Key type lgssc unregistered [ 7647.717614] LNet: 224357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7647.722894] LNetError: 224357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7648.751990] LNet: Removed LNI 192.168.201.146@tcp [ 7649.940165] Key type .llcrypt unregistered [ 7649.944107] Key type ._llcrypt unregistered [ 7697.891885] Key type ._llcrypt registered [ 7697.894956] Key type .llcrypt registered [ 7698.010393] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 7699.965126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7699.982500] alg: No test for adler32 (adler32-zlib) [ 7701.138653] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 7701.364957] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 7703.088223] Key type lgssc registered [ 7704.695886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7713.564671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7713.602034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7715.038748] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7715.074833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7715.264808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7719.911451] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7728.843139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7729.014923] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7729.068635] Lustre: 226565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7729.315603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7734.346769] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7742.474089] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7750.520990] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7759.957159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7760.357409] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7760.829784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7768.181049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7769.873873] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7778.775855] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7783.033958] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7790.025223] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7798.284865] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7804.358663] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7821.688523] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7821.694482] Lustre: Skipped 1 previous similar message [ 7839.681572] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7839.687455] Lustre: Skipped 2 previous similar messages [ 7874.197149] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7874.200689] Lustre: Skipped 3 previous similar messages [ 7946.594374] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7946.602641] Lustre: Skipped 8 previous similar messages [ 7962.602306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7962.618663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7962.664508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7963.106325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7963.119594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7964.596238] Lustre: server umount lustre-OST0000 complete [ 7970.284517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7970.306154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7970.350702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7970.377782] Lustre: Skipped 2 previous similar messages [ 7973.346232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7973.346927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7975.657258] Lustre: server umount lustre-MDT0000 complete [ 7980.913649] LustreError: 229353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071704 with bad export cookie 9016029401723737297 [ 7980.914698] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7980.926801] LustreError: 229353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7981.354314] Lustre: server umount lustre-MDT0001 complete [ 7994.729116] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7998.505597] Key type lgssc unregistered [ 7998.872835] LNet: 231387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7998.879906] LNetError: 231387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7998.900537] LNet: Removed LNI 192.168.201.146@tcp [ 7999.835876] Key type .llcrypt unregistered [ 7999.839100] Key type ._llcrypt unregistered [ 8029.451418] Key type ._llcrypt registered [ 8029.458860] Key type .llcrypt registered [ 8029.639322] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 8031.080076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8031.322989] alg: No test for adler32 (adler32-zlib) [ 8032.439383] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 8032.983032] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 8034.744233] Key type lgssc registered [ 8036.527838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8045.986539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8046.030672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8047.635859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8052.319386] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8056.823470] LustreError: 232797:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8056.866120] LustreError: 232797:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8061.927358] LustreError: 232796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8061.959580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8062.595724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8068.591032] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8080.803702] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8091.121935] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8099.700584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8100.015076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8107.270295] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8117.441309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 8118.015492] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8138.219117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8138.220760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8138.235995] LustreError: Skipped 1 previous similar message [ 8138.254048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8138.744479] Lustre: server umount lustre-OST0000 complete [ 8144.357417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8144.372984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8144.398259] Lustre: Skipped 1 previous similar message [ 8144.407336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8144.429154] Lustre: Skipped 1 previous similar message [ 8146.401405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8146.401901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8149.134359] Lustre: server umount lustre-MDT0000 complete [ 8151.528876] LustreError: 232797:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8151.544064] LustreError: 232797:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8153.655846] LustreError: 232777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071876 with bad export cookie 16242532333493967119 [ 8153.657444] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8153.668355] LustreError: 232777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8154.069663] Lustre: server umount lustre-MDT0001 complete [ 8168.023941] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8171.015044] Key type lgssc unregistered [ 8171.364685] LNet: 236485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8171.375862] LNetError: 236485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8171.405796] LNet: Removed LNI 192.168.201.146@tcp [ 8172.575541] Key type .llcrypt unregistered [ 8172.577339] Key type ._llcrypt unregistered [ 8200.536742] Key type ._llcrypt registered [ 8200.539791] Key type .llcrypt registered [ 8200.664649] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 8201.864910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8201.927928] alg: No test for adler32 (adler32-zlib) [ 8203.257599] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 8203.521223] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 8205.176347] Key type lgssc registered [ 8206.567556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8477.734343] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8480.289990] Key type lgssc unregistered [ 8480.571687] LNet: 238504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8480.581418] LNetError: 238504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8480.599091] LNet: Removed LNI 192.168.201.146@tcp [ 8481.633697] Key type .llcrypt unregistered [ 8481.635713] Key type ._llcrypt unregistered [ 8509.872895] Key type ._llcrypt registered [ 8509.875919] Key type .llcrypt registered [ 8510.019211] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 8533.800700] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 8535.838246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8535.957626] alg: No test for adler32 (adler32-zlib) [ 8537.567405] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 8538.030949] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 8539.868186] Key type lgssc registered [ 8541.815894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8580.568802] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8214 sec ========== 16:31:43 (1789072303) [ 8582.659834] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:31:44 (1789072304) === [ 8586.253942] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:31:48 (1789072308) === [ 8638.168849] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8642.478721] Key type lgssc unregistered [ 8642.830196] LNet: 243761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8642.842499] LNetError: 243761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8642.860712] LNet: Removed LNI 192.168.201.146@tcp [ 8644.111331] Key type .llcrypt unregistered [ 8644.116354] Key type ._llcrypt unregistered