[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 529839774 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003288] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010021] pid_max: default: 32768 minimum: 301 [ 0.011147] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.015037] *** VALIDATE selinux *** [ 0.023839] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028887] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032105] *** VALIDATE tmpfs *** [ 0.033480] *** VALIDATE proc *** [ 0.035029] *** VALIDATE cgroup *** [ 0.036010] *** VALIDATE cgroup2 *** [ 0.037292] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039082] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041031] Spectre V2 : User space: Vulnerable [ 0.042008] Speculative Store Bypass: Vulnerable [ 0.045289] debug: unmapping init [mem 0xffffffffa9259000-0xffffffffa9260fff] [ 0.047930] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048703] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050013] ... bit width: 48 [ 0.051012] ... generic registers: 4 [ 0.052012] ... value mask: 0000ffffffffffff [ 0.053018] ... max period: 00007fffffffffff [ 0.054015] ... fixed-purpose events: 3 [ 0.055011] ... event mask: 000000070000000f [ 0.056321] rcu: Hierarchical SRCU implementation. [ 0.058505] smp: Bringing up secondary CPUs ... [ 0.059572] x86: Booting SMP configuration: [ 0.060028] .... node #0, CPUs: #1 #2 #3 [ 0.068102] smp: Brought up 1 node, 4 CPUs [ 0.070020] smpboot: Max logical packages: 1 [ 0.071016] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.183424] node 0 deferred pages initialised in 110ms [ 0.187186] devtmpfs: initialized [ 0.188231] x86/mm: Memory block size: 128MB [ 0.190808] gcov: version magic: 0x41383552 [ 0.192051] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.195076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.198361] pinctrl core: initialized pinctrl subsystem [ 0.200192] [ 0.200951] ************************************************************* [ 0.203014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.205010] ** ** [ 0.207014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.209013] ** ** [ 0.211013] ** This means that this kernel is built to expose internal ** [ 0.214016] ** IOMMU data structures, which may compromise security on ** [ 0.216012] ** your system. ** [ 0.218014] ** ** [ 0.220014] ** If you see this message and you are not debugging the ** [ 0.222017] ** kernel, report this immediately to your vendor! ** [ 0.224012] ** ** [ 0.226012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.228012] ************************************************************* [ 0.230629] NET: Registered protocol family 16 [ 0.232444] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.234073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.237057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.240116] cpuidle: using governor menu [ 0.242000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.244557] PCI: Using configuration type 1 for base access [ 0.246133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.257157] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.260132] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.265103] cryptd: max_cpu_qlen set to 1000 [ 0.270150] ACPI: Added _OSI(Module Device) [ 0.272027] ACPI: Added _OSI(Processor Device) [ 0.274023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.276022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.280845] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.289055] ACPI: Interpreter enabled [ 0.291088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.292016] ACPI: Using IOAPIC for interrupt routing [ 0.294173] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.298441] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.309478] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.312055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.315032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.320076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.325630] acpiphp: Slot [2] registered [ 0.327204] acpiphp: Slot [5] registered [ 0.329197] acpiphp: Slot [6] registered [ 0.330171] acpiphp: Slot [7] registered [ 0.332199] acpiphp: Slot [8] registered [ 0.334206] acpiphp: Slot [9] registered [ 0.337180] acpiphp: Slot [10] registered [ 0.338135] acpiphp: Slot [3] registered [ 0.339122] acpiphp: Slot [4] registered [ 0.341093] acpiphp: Slot [11] registered [ 0.342099] acpiphp: Slot [12] registered [ 0.343137] acpiphp: Slot [13] registered [ 0.345178] acpiphp: Slot [14] registered [ 0.346129] acpiphp: Slot [15] registered [ 0.348133] acpiphp: Slot [16] registered [ 0.351364] acpiphp: Slot [17] registered [ 0.353318] acpiphp: Slot [18] registered [ 0.355203] acpiphp: Slot [19] registered [ 0.358185] acpiphp: Slot [20] registered [ 0.360339] acpiphp: Slot [21] registered [ 0.362145] acpiphp: Slot [22] registered [ 0.364153] acpiphp: Slot [23] registered [ 0.366287] acpiphp: Slot [24] registered [ 0.368257] acpiphp: Slot [25] registered [ 0.370158] acpiphp: Slot [26] registered [ 0.372114] acpiphp: Slot [27] registered [ 0.373255] acpiphp: Slot [28] registered [ 0.375213] acpiphp: Slot [29] registered [ 0.377265] acpiphp: Slot [30] registered [ 0.379130] acpiphp: Slot [31] registered [ 0.381079] PCI host bridge to bus 0000:00 [ 0.382082] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.385023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.388031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.391028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.394034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.397038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.399231] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.404764] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.408653] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.421014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.426563] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.429029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.432034] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.436034] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.439425] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.443935] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.447048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.450800] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.457017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.472028] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.477015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.483869] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.498029] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.506023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.521029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.531274] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.540020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.548017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.576019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.590272] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.598024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.606024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.627021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.638727] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.646019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.654016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.677021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.691387] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.699028] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.710026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.732031] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.751600] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.761026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.769023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.789023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.804515] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.807548] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.811546] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.814556] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.817355] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.823179] iommu: Default domain type: Passthrough [ 0.824805] SCSI subsystem initialized [ 0.825259] ACPI: bus type USB registered [ 0.826198] usbcore: registered new interface driver usbfs [ 0.827125] usbcore: registered new interface driver hub [ 0.830193] usbcore: registered new device driver usb [ 0.832306] pps_core: LinuxPPS API ver. 1 registered [ 0.835018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.839091] PTP clock support registered [ 0.842174] EDAC MC: Ver: 3.0.0 [ 0.844198] PCI: Using ACPI for IRQ routing [ 0.847051] NetLabel: Initializing [ 0.849015] NetLabel: domain hash size = 128 [ 0.851014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.853139] NetLabel: unlabeled traffic allowed by default [ 0.856052] vgaarb: loaded [ 0.857328] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.860024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.866011] clocksource: Switched to clocksource kvm-clock [ 0.994441] VFS: Disk quotas dquot_6.6.0 [ 0.996445] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.999383] *** VALIDATE ramfs *** [ 1.000700] *** VALIDATE hugetlbfs *** [ 1.002467] pnp: PnP ACPI init [ 1.005022] pnp: PnP ACPI: found 6 devices [ 1.025311] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.029381] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.031801] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.034649] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.037962] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.041437] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.045347] NET: Registered protocol family 2 [ 1.048887] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.054993] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.058888] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.064756] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.068868] TCP: Hash tables configured (established 65536 bind 65536) [ 1.072294] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.075626] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.078775] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.081971] NET: Registered protocol family 1 [ 1.085916] RPC: Registered named UNIX socket transport module. [ 1.088349] RPC: Registered udp transport module. [ 1.090299] RPC: Registered tcp transport module. [ 1.092286] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.094938] NET: Registered protocol family 44 [ 1.096931] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.099725] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.102197] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.106955] PCI: CLS 0 bytes, default 64 [ 1.109746] Unpacking initramfs... [ 2.517386] debug: unmapping init [mem 0xffff8882fcc54000-0xffff8882fffbffff] [ 2.521320] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.523217] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.526722] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.027419] Initialise system trusted keyrings [ 3.031042] Key type blacklist registered [ 3.033543] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.042208] zbud: loaded [ 3.046713] *** VALIDATE nfs *** [ 3.048312] *** VALIDATE nfs4 *** [ 3.049837] pstore: using deflate compression [ 3.055866] Platform Keyring initialized [ 3.165992] NET: Registered protocol family 38 [ 3.168376] Key type asymmetric registered [ 3.170166] Asymmetric key parser 'x509' registered [ 3.172703] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.176294] io scheduler mq-deadline registered [ 3.178600] io scheduler kyber registered [ 3.180651] io scheduler bfq registered [ 3.183259] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.186922] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.191886] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.195909] ACPI: Power Button [PWRF] [ 3.202809] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.211458] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.225698] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.237596] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.258282] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.288716] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.317693] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.322514] Non-volatile memory driver v1.3 [ 3.324165] Linux agpgart interface v0.103 [ 3.358664] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.363162] vda: detected capacity change from 0 to 74702848 [ 3.386481] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.390866] vdb: detected capacity change from 0 to 1073741824 [ 3.409881] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.413731] vdc: detected capacity change from 0 to 2621440000 [ 3.432595] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.435914] vdd: detected capacity change from 0 to 2621440000 [ 3.454197] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.457555] vde: detected capacity change from 0 to 4294967296 [ 3.473409] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.476491] vdf: detected capacity change from 0 to 4294967296 [ 3.485867] libphy: Fixed MDIO Bus: probed [ 3.493141] usbcore: registered new interface driver usbserial_generic [ 3.495345] usbserial: USB Serial support registered for generic [ 3.496939] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.500149] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.502680] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.505478] mousedev: PS/2 mouse device common for all mice [ 3.508934] rtc_cmos 00:05: RTC can wake from S4 [ 3.512518] rtc_cmos 00:05: registered as rtc0 [ 3.512612] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.514446] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.521700] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.522882] intel_pstate: CPU model not supported [ 3.529569] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.530211] hid: raw HID events driver (C) Jiri Kosina [ 3.535382] usbcore: registered new interface driver usbhid [ 3.537478] usbhid: USB HID core driver [ 3.539161] drop_monitor: Initializing network drop monitor service [ 3.541921] Initializing XFRM netlink socket [ 3.545304] NET: Registered protocol family 10 [ 3.548717] Segment Routing with IPv6 [ 3.550276] NET: Registered protocol family 17 [ 3.552475] mpls_gso: MPLS GSO support [ 3.558809] RAS: Correctable Errors collector initialized. [ 3.561192] AVX version of gcm_enc/dec engaged. [ 3.562204] AES CTR mode by8 optimization enabled [ 3.660089] sched_clock: Marking stable (3660031248, 0)->(4591244347, -931213099) [ 3.663926] registered taskstats version 1 [ 3.666414] Loading compiled-in X.509 certificates [ 3.668722] zswap: loaded using pool lzo/zbud [ 3.705359] Key type big_key registered [ 3.721032] Key type encrypted registered [ 3.723223] ima: No TPM chip found, activating TPM-bypass! [ 3.726137] ima: Allocated hash algorithm: sha1 [ 3.728806] ima: No architecture policies found [ 3.730937] evm: Initialising EVM extended attributes: [ 3.733134] evm: security.selinux [ 3.734604] evm: security.ima [ 3.735794] evm: security.capability [ 3.737459] evm: HMAC attrs: 0x1 [ 3.740476] rtc_cmos 00:05: setting system clock to 2026-08-15 16:37:10 UTC (1786811830) [ 3.749299] debug: unmapping init [mem 0xffffffffaa203000-0xffffffffaa3fffff] [ 3.753020] debug: unmapping init [mem 0xffffffffa8f82000-0xffffffffa9258fff] [ 3.759093] Write protecting the kernel read-only data: 28672k [ 3.762483] debug: unmapping init [mem 0xffffffffa7603000-0xffffffffa77fffff] [ 3.765545] debug: unmapping init [mem 0xffffffffa7f14000-0xffffffffa7ffffff] [ 3.814724] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.825418] systemd[1]: Detected virtualization kvm. [ 3.827248] systemd[1]: Detected architecture x86-64. [ 3.829558] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.855366] systemd[1]: No hostname configured. [ 3.856713] systemd[1]: Set hostname to . [ 3.858926] random: systemd: uninitialized urandom read (16 bytes read) [ 3.861770] systemd[1]: Initializing machine ID from random generator. [ 4.023611] random: systemd: uninitialized urandom read (16 bytes read) [ 4.026850] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.033234] random: systemd: uninitialized urandom read (16 bytes read) [ 4.037444] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.043227] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.719685] device-mapper: uevent: version 1.0.3 [ 4.722481] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.511464] virtio_net virtio0 ens2: renamed from eth0 [ 5.522126] random: fast init done [ 5.589243] scsi host0: ata_piix [ 5.613415] scsi host1: ata_piix [ 5.614912] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.617490] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.394737] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.243803] random: crng init done [ 10.245569] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.772715] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.019143] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.309207] SELinux: Disabled at runtime. [ 12.375060] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.385960] systemd[1]: Detected virtualization kvm. [ 12.388578] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.992026] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.995777] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.001126] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.006302] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.010213] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.019276] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.026376] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ 13.132858] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.522560] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.858196] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.917995] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.014215] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.032048] EDAC sbridge: Ver: 1.1.2 [ 15.862321] Key type dns_resolver registered [ 16.190994] NFS: Registering the id_resolver key type [ 16.193834] Key type id_resolver registered [ 16.196290] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg417-server login: [ 49.968629] libcfs: loading out-of-tree module taints kernel. [ 50.033556] Key type ._llcrypt registered [ 50.038930] Key type .llcrypt registered [ 50.149262] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 70.895860] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 72.424245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 72.437649] alg: No test for adler32 (adler32-zlib) [ 73.803608] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 74.633483] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 76.393591] Key type lgssc registered [ 78.820432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.348621] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 138.185657] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 149.848223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 149.885053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 151.210710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 151.246836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 151.329981] Lustre: lustre-MDT0000: new disk, initializing [ 151.454255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 151.487287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 153.908133] hrtimer: interrupt took 6122471 ns [ 155.510707] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.172852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.354878] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 167.407719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 167.413389] Lustre: Skipped 1 previous similar message [ 167.562285] Lustre: lustre-MDT0001: new disk, initializing [ 167.668342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 167.726533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 167.743722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 172.873729] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.546196] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 191.992925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 192.573028] Lustre: lustre-OST0000: new disk, initializing [ 192.583437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 192.598047] Lustre: 8447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.748915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.206642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 199.235169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 199.317208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.622484] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.441723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.679277] Lustre: lustre-OST0001: new disk, initializing [ 216.684663] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.690799] Lustre: 9520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.779569] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.793888] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.352925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 226.360479] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 226.399227] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 234.630502] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 241.824880] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 248.951038] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing check_logdir /tmp/testlogs/ [ 255.677368] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing yml_node [ 260.320837] Lustre: DEBUG MARKER: Client: 2.17.57.3 [ 263.583661] Lustre: DEBUG MARKER: MDS: 2.17.57.3 [ 266.150762] Lustre: DEBUG MARKER: OSS: 2.17.57.3 [ 268.152624] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 12:41:32 EDT 2026 [ 283.937142] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 285.615220] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 301.025115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 301.036326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.046235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.090341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.095961] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.121390] Lustre: Skipped 2 previous similar messages [ 307.036275] Lustre: server umount lustre-MDT0000 complete [ 310.423741] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812137 with bad export cookie 5769797330147256939 [ 310.427071] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 310.434667] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 313.313685] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.318825] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.319404] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.319409] Lustre: Skipped 3 previous similar messages [ 313.323646] Lustre: Skipped 1 previous similar message [ 313.366800] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 315.363931] LustreError: 9155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 315.364201] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 315.386645] LustreError: 9155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 315.424900] Lustre: Skipped 2 previous similar messages [ 316.756443] Lustre: server umount lustre-MDT0001 complete [ 320.590396] Lustre: server umount lustre-OST0000 complete [ 323.986096] Lustre: server umount lustre-OST0001 complete [ 331.310845] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 342.648260] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 380.062563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.296383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 380.325666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 380.421854] Lustre: lustre-MDT0000: new disk, initializing [ 380.494509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 380.511751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 385.032581] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 395.395901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.583817] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 395.718881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 395.731359] Lustre: Skipped 1 previous similar message [ 395.886494] Lustre: lustre-MDT0001: new disk, initializing [ 396.072818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 396.103461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 396.112324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 400.594269] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 409.844721] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 416.286084] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 424.603392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 424.991439] Lustre: lustre-OST0000: new disk, initializing [ 425.014882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 425.026741] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 425.152596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.328333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 426.336549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 426.408434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 431.998279] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 441.739457] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 447.226795] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 447.433842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 453.170891] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 453.434800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 457.200712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.202174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.224266] LustreError: Skipped 1 previous similar message [ 457.244019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.254707] Lustre: Skipped 1 previous similar message [ 462.542528] Lustre: server umount lustre-OST0000 complete [ 467.426326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.449168] Lustre: Skipped 1 previous similar message [ 467.455261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.466423] Lustre: Skipped 3 previous similar messages [ 473.079298] Lustre: server umount lustre-MDT0000 complete [ 476.793102] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812303 with bad export cookie 5769797330147258472 [ 476.808543] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 476.809845] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 477.232350] Lustre: server umount lustre-MDT0001 complete [ 480.143533] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 12:45:04 (1786812304) [ 486.700717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 487.151183] LustreError: 19309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.223994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.357550] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 492.525435] LustreError: 19310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.614353] LustreError: 19309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.975653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.432026] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 519.356898] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 526.154602] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 532.762823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 538.940552] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 546.039814] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 567.266627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.287734] Lustre: Skipped 1 previous similar message [ 567.294815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 567.303915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 567.314222] Lustre: Skipped 3 previous similar messages [ 570.057451] Lustre: server umount lustre-OST0000 complete [ 574.434162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 579.890683] Lustre: server umount lustre-MDT0000 complete [ 582.634266] LustreError: 19310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.646959] LustreError: 19310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 583.376500] LustreError: 19289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812410 with bad export cookie 5769797330147259354 [ 583.396501] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 583.629283] Lustre: server umount lustre-MDT0001 complete [ 594.018362] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 596.925672] Key type lgssc unregistered [ 597.225822] LNet: 22866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.234277] LNetError: 22866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 597.247920] LNet: Removed LNI 192.168.204.117@tcp [ 598.041297] Key type .llcrypt unregistered [ 598.043347] Key type ._llcrypt unregistered [ 618.156970] Key type ._llcrypt registered [ 618.159189] Key type .llcrypt registered [ 618.258100] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 619.416155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 619.454235] alg: No test for adler32 (adler32-zlib) [ 620.557630] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 620.784688] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 622.472554] Key type lgssc registered [ 623.946664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 630.869463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 630.890816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 632.513142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 636.631844] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 643.048217] LustreError: 24258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.078166] LustreError: 24258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 644.588125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.043817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 649.403782] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 656.791731] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 661.956584] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 669.215763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 669.725123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 675.768793] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 678.024175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 681.929895] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 698.852106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 698.854863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 698.874699] Lustre: Skipped 1 previous similar message [ 698.887869] Lustre: Skipped 1 previous similar message [ 699.814610] Lustre: server umount lustre-OST0000 complete [ 703.970939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 704.012024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 704.023190] Lustre: Skipped 1 previous similar message [ 709.097310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.110597] Lustre: Skipped 1 previous similar message [ 709.883721] Lustre: server umount lustre-MDT0000 complete [ 713.548450] LustreError: 24239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812540 with bad export cookie 15302900756340391439 [ 713.566754] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 713.852896] Lustre: server umount lustre-MDT0001 complete [ 725.519508] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 728.113839] Key type lgssc unregistered [ 728.569608] LNet: 28236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.576873] LNetError: 28236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.595284] LNet: Removed LNI 192.168.204.117@tcp [ 729.956864] Key type .llcrypt unregistered [ 729.962673] Key type ._llcrypt unregistered [ 753.768739] Key type ._llcrypt registered [ 753.771572] Key type .llcrypt registered [ 753.881537] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 755.130806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 755.142927] alg: No test for adler32 (adler32-zlib) [ 756.127667] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 756.309742] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 757.944181] Key type lgssc registered [ 759.479423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 768.712832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 768.740548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 770.268197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 774.924993] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 779.751458] LustreError: 29649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 779.772856] LustreError: 29649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 783.035322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 783.418330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 786.940544] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 793.675096] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 799.064186] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 811.105119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 811.473345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 814.584265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 818.023985] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 826.581109] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 835.041351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 835.045750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 835.052658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 839.137734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 839.142742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 839.157960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 840.510810] Lustre: server umount lustre-OST0000 complete [ 844.257468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.261881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 844.288480] Lustre: Skipped 1 previous similar message [ 844.308648] Lustre: Skipped 2 previous similar messages [ 849.377942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 849.390092] Lustre: Skipped 1 previous similar message [ 850.350889] Lustre: server umount lustre-MDT0000 complete [ 854.502913] LustreError: 29649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 855.867066] LustreError: 29631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812682 with bad export cookie 10512990670861162170 [ 855.902325] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 855.907830] LustreError: 29631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 856.474472] Lustre: server umount lustre-MDT0001 complete [ 869.299945] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 872.771943] Key type lgssc unregistered [ 873.087584] LNet: 33535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.092072] LNetError: 33535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.114585] LNet: Removed LNI 192.168.204.117@tcp [ 874.168408] Key type .llcrypt unregistered [ 874.172978] Key type ._llcrypt unregistered [ 899.989740] Key type ._llcrypt registered [ 899.995118] Key type .llcrypt registered [ 900.081550] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 901.197864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 901.234830] alg: No test for adler32 (adler32-zlib) [ 902.663802] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 902.946266] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 904.648775] Key type lgssc registered [ 906.144319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 915.338213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 915.372972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.893902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 921.100265] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 927.201253] LustreError: 34948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.230078] LustreError: 34948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 929.492264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 929.898564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 934.878161] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 941.736712] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 946.980646] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 953.502596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 953.902781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.568829] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 962.167585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 970.318246] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 983.036432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.037776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 983.045658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 983.062982] Lustre: Skipped 1 previous similar message [ 987.267156] Lustre: server umount lustre-OST0000 complete [ 993.252775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.260918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 993.265043] Lustre: Skipped 2 previous similar messages [ 993.274808] Lustre: Skipped 1 previous similar message [ 998.370398] LustreError: 34948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.380848] LustreError: 34948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 998.435605] Lustre: server umount lustre-MDT0000 complete [ 1002.468521] LustreError: 34943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1002.477990] LustreError: 34943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1002.730349] LustreError: 34929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812829 with bad export cookie 10895562412097129450 [ 1002.731959] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.744419] LustreError: 34929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1002.984748] Lustre: server umount lustre-MDT0001 complete [ 1014.848859] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 1017.518741] Key type lgssc unregistered [ 1017.707825] LNet: 38484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.718690] LNetError: 38484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.728862] LNet: Removed LNI 192.168.204.117@tcp [ 1018.413577] Key type .llcrypt unregistered [ 1018.416391] Key type ._llcrypt unregistered [ 1041.921301] Key type ._llcrypt registered [ 1041.924477] Key type .llcrypt registered [ 1042.040392] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 1043.092676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1043.226750] alg: No test for adler32 (adler32-zlib) [ 1044.269305] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 1044.472373] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 1046.160206] Key type lgssc registered [ 1047.283202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1055.041095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1055.055978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1056.776771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1061.699910] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1066.982712] LustreError: 39898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1067.025932] LustreError: 39898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1071.677140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.095482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1076.430162] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1084.428948] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1091.190406] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1099.459544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1099.962613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.012518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1107.385723] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1115.073252] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1121.763681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.776184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1121.781032] Lustre: Skipped 1 previous similar message [ 1126.500545] Lustre: server umount lustre-OST0000 complete [ 1147.365446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1147.369117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1147.386286] Lustre: Skipped 2 previous similar messages [ 1147.404979] Lustre: Skipped 2 previous similar messages [ 1152.485076] LustreError: 39898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1152.497257] LustreError: 39898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1152.647218] Lustre: server umount lustre-MDT0000 complete [ 1156.651499] LustreError: 39879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812983 with bad export cookie 13759986311869696039 [ 1156.671748] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1163.112271] Lustre: server umount lustre-MDT0001 complete [ 1173.806820] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 1176.853417] Key type lgssc unregistered [ 1177.231513] LNet: 43611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.241364] LNetError: 43611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.282253] LNet: Removed LNI 192.168.204.117@tcp [ 1178.160630] Key type .llcrypt unregistered [ 1178.163322] Key type ._llcrypt unregistered [ 1202.870975] Key type ._llcrypt registered [ 1202.873301] Key type .llcrypt registered [ 1202.965738] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 1204.185585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1204.462783] alg: No test for adler32 (adler32-zlib) [ 1205.606994] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 1205.931918] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 1207.824309] Key type lgssc registered [ 1209.206494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1218.674478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1218.709036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.579753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.132058] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1230.838398] LustreError: 45025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.858400] LustreError: 45025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1235.518435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.945832] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1236.310182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1242.348271] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1253.321351] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1260.204343] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1267.361301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1267.716874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1274.362865] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1280.960382] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1291.290261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1297.889024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1297.894354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.911948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1299.766642] Lustre: server umount lustre-MDT0000 complete [ 1301.477130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.479037] LustreError: 45026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1301.527123] LustreError: 45026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1303.336057] LustreError: 47386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.404185] LustreError: 45007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813130 with bad export cookie 1545575256784482362 [ 1303.418895] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1306.604751] LustreError: 45022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.605468] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1306.637098] Lustre: Skipped 1 previous similar message [ 1306.643864] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1306.654237] Lustre: Skipped 1 previous similar message [ 1308.645747] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1308.655790] Lustre: Skipped 1 previous similar message [ 1309.835740] Lustre: server umount lustre-MDT0001 complete [ 1339.361704] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1339.584756] Lustre: server umount lustre-OST0000 complete [ 1355.757479] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 1358.679428] Key type lgssc unregistered [ 1358.912236] LNet: 48914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1358.943244] LNetError: 48914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1358.970191] LNet: Removed LNI 192.168.204.117@tcp [ 1359.907733] Key type .llcrypt unregistered [ 1359.911652] Key type ._llcrypt unregistered [ 1387.175223] Key type ._llcrypt registered [ 1387.182993] Key type .llcrypt registered [ 1387.329478] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 1388.663130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1389.174551] alg: No test for adler32 (adler32-zlib) [ 1390.477553] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 1390.884671] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 1392.651085] Key type lgssc registered [ 1394.799063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1406.967902] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1406.982339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1422.496368] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1427.617380] LustreError: 50308:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.117@tcp: failed processing log, type 4: rc = -110 [ 1454.304268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.716545] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1467.748791] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1635.526657] Lustre: server umount lustre-OST0000 complete [ 1652.329192] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 1655.092304] Key type lgssc unregistered [ 1655.533647] LNet: 51999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.544636] LNetError: 51999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1656.619027] LNet: Removed LNI 192.168.204.117@tcp [ 1657.746854] Key type .llcrypt unregistered [ 1657.748914] Key type ._llcrypt unregistered [ 1680.027032] Key type ._llcrypt registered [ 1680.032200] Key type .llcrypt registered [ 1680.129740] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 1681.084024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1681.117461] alg: No test for adler32 (adler32-zlib) [ 1682.362721] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 1682.611699] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 1684.280208] Key type lgssc registered [ 1685.265586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1694.310609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1694.340039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1696.272883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.325135] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1705.445973] LustreError: 53413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1705.500929] LustreError: 53413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1710.562763] LustreError: 53414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1713.141584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1713.569311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1718.711254] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1728.859984] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1735.659190] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1742.779354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1743.208987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.985344] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1758.005904] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1763.817775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1763.848252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1765.927868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1767.384596] Lustre: server umount lustre-OST0000 complete [ 1775.075396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1775.084082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.100213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1775.103533] Lustre: Skipped 1 previous similar message [ 1776.117060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1778.892540] Lustre: server umount lustre-MDT0000 complete [ 1781.218226] LustreError: 53410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1781.249422] LustreError: 53410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1783.868384] LustreError: 53393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813610 with bad export cookie 4119475382621702857 [ 1783.868735] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1783.879039] LustreError: 53393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1784.392211] Lustre: server umount lustre-MDT0001 complete [ 1797.319224] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 1801.386162] Key type lgssc unregistered [ 1801.777642] LNet: 56949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.783671] LNetError: 56949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.795209] LNet: Removed LNI 192.168.204.117@tcp [ 1802.857535] Key type .llcrypt unregistered [ 1802.860516] Key type ._llcrypt unregistered [ 1830.361620] Key type ._llcrypt registered [ 1830.364873] Key type .llcrypt registered [ 1830.523147] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 1832.038484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1832.066255] alg: No test for adler32 (adler32-zlib) [ 1833.219450] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 1833.534250] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 1835.192225] Key type lgssc registered [ 1836.673935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1848.667670] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1848.698424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1864.353270] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1869.472311] LustreError: 58336:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.117@tcp: failed processing log, type 4: rc = -110 [ 1896.160805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.238406] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1909.653947] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1915.655085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1916.321018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.397511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1920.630903] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1921.507158] LustreError: 58362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1921.527708] LustreError: 58362:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1926.629030] LustreError: 58361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1928.928964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1929.267713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1933.056355] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1940.246764] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1945.149350] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1947.107507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1947.122707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.134761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1949.666254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1949.674397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.705666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1953.397440] Lustre: server umount lustre-OST0000 complete [ 1973.064760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.071488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1973.075702] Lustre: Skipped 1 previous similar message [ 1973.086873] Lustre: Skipped 3 previous similar messages [ 1976.102865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1976.115105] Lustre: Skipped 1 previous similar message [ 1979.511662] Lustre: server umount lustre-MDT0000 complete [ 1982.440277] LustreError: 59377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.491260] LustreError: 59377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1983.349456] LustreError: 58344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813810 with bad export cookie 8320864780799523384 [ 1983.361290] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1983.365216] LustreError: 58344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1989.811448] Lustre: server umount lustre-MDT0001 complete [ 2001.540981] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 2005.181084] Key type lgssc unregistered [ 2005.456696] LNet: 62085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.469770] LNetError: 62085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.512421] LNet: Removed LNI 192.168.204.117@tcp [ 2007.559300] Key type .llcrypt unregistered [ 2007.561862] Key type ._llcrypt unregistered [ 2032.091655] Key type ._llcrypt registered [ 2032.097440] Key type .llcrypt registered [ 2032.249637] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 2033.914301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2034.074885] alg: No test for adler32 (adler32-zlib) [ 2035.114201] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 2035.291391] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 2036.931214] Key type lgssc registered [ 2037.941530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2045.200298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2045.218589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.723784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.091570] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2057.194143] LustreError: 63499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2057.222304] LustreError: 63499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2060.462699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.088842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2067.262270] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2076.078611] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2081.943435] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2089.284858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2089.775314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.855630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2096.077658] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2104.737456] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2130.752834] Lustre: server umount lustre-OST0000 complete [ 2137.058776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.066953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.088060] Lustre: Skipped 1 previous similar message [ 2142.181065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2142.192985] Lustre: Skipped 2 previous similar messages [ 2147.296219] Lustre: 62812:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786813958/real 1786813958] req@000000003da409a0 x1873610132192512/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786813974 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2147.298531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2147.331607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.359192] Lustre: Skipped 1 previous similar message [ 2149.344278] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2149.544996] Lustre: server umount lustre-MDT0000 complete [ 2152.417681] LustreError: 63499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2152.422715] Lustre: 62813:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786813963/real 1786813963] req@000000004f0458fb x1873610132195072/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786813979 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2152.461329] LustreError: 63499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2153.857323] LustreError: 63480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813980 with bad export cookie 17620454661303159373 [ 2153.869338] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2153.877107] LustreError: 63480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2154.384053] Lustre: server umount lustre-MDT0001 complete [ 2164.786754] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 2167.176908] Key type lgssc unregistered [ 2167.413825] LNet: 67063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.422775] LNetError: 67063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2167.434602] LNet: Removed LNI 192.168.204.117@tcp [ 2168.165584] Key type .llcrypt unregistered [ 2168.168113] Key type ._llcrypt unregistered [ 2193.295428] Key type ._llcrypt registered [ 2193.299411] Key type .llcrypt registered [ 2193.419174] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 2194.571403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2194.619076] alg: No test for adler32 (adler32-zlib) [ 2195.637597] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 2195.863569] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 2197.512235] Key type lgssc registered [ 2198.637491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2207.363469] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 13:13:51 (1786814031) [ 2213.612480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2213.642835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.998729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2218.709902] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2225.126602] LustreError: 68768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.152775] LustreError: 68768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2226.513705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2226.924422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2231.130857] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2238.101827] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2243.278302] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2249.119175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2249.348474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.697280] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2259.636593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2260.797363] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2266.455929] Lustre: Failing over lustre-MDT0000 [ 2266.724078] Lustre: server umount lustre-MDT0000 complete [ 2268.137692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2268.142938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.166209] LustreError: 68768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.178853] LustreError: 68767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.186068] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2270.193736] LustreError: 68767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2270.199441] Lustre: Skipped 1 previous similar message [ 2273.568729] LustreError: 68764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.591405] LustreError: 68764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2274.196903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2274.347378] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2274.459652] Lustre: *** cfs_fail_loc=135, val=0*** [ 2274.540207] LustreError: 71851:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2274.545897] LustreError: 71851:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.117@tcp: cfg command failed: rc = -2 [ 2274.560022] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2274.560022] [ 2274.569264] LustreError: MGC192.168.204.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2274.581345] LustreError: 71840:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2274.596807] LustreError: 71840:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2274.609414] LustreError: 71840:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2274.619453] LustreError: 71840:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2274.638796] Lustre: server umount lustre-MDT0000 complete [ 2274.646190] LustreError: 71840:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2278.701077] LustreError: 70728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.714232] LustreError: 70728:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2280.771417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.950652] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2280.958228] LustreError: 68760:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2281.238523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2281.311123] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2283.817464] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2285.829476] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2286.582224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2286.617244] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2286.693728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2296.802698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2296.811491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2296.812412] Lustre: Skipped 1 previous similar message [ 2296.838030] Lustre: Skipped 1 previous similar message [ 2299.384481] Lustre: server umount lustre-OST0000 complete [ 2307.045632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.048518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2307.063145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2307.073974] Lustre: Skipped 1 previous similar message [ 2309.499532] Lustre: server umount lustre-MDT0000 complete [ 2312.162571] LustreError: 68767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2312.189085] LustreError: 68767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2312.545471] LustreError: 68750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814139 with bad export cookie 2436497260124091964 [ 2312.547608] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2312.553673] LustreError: 68750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2312.827743] Lustre: server umount lustre-MDT0001 complete [ 2321.752349] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 2324.038638] Key type lgssc unregistered [ 2324.277720] LNet: 73632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.289774] LNetError: 73632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.316138] LNet: Removed LNI 192.168.204.117@tcp [ 2325.022324] Key type .llcrypt unregistered [ 2325.024151] Key type ._llcrypt unregistered [ 2345.687948] Key type ._llcrypt registered [ 2345.691519] Key type .llcrypt registered [ 2345.802974] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 2346.644729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2346.661378] alg: No test for adler32 (adler32-zlib) [ 2347.746919] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 2348.017653] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 2349.657048] Key type lgssc registered [ 2350.548523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2356.792790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2356.827675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.255445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.432628] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2367.471816] LustreError: 75034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2367.489058] LustreError: 75034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2370.667728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.962160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2374.913105] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2382.079667] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2388.991880] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2398.428191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.841454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.950099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2405.890740] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2413.426764] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2422.803566] Lustre: Failing over lustre-MDT0000 [ 2423.108111] Lustre: server umount lustre-MDT0000 complete [ 2426.199466] LustreError: 75029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.855224] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.871223] Lustre: Skipped 1 previous similar message [ 2431.268183] LustreError: 77552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.299135] LustreError: 77552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2433.170215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.257690] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2433.344763] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2433.453169] LustreError: 78124:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2433.460796] LustreError: 78124:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.117@tcp: cfg command failed: rc = -9 [ 2433.475551] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2433.475551] [ 2433.486356] LustreError: MGC192.168.204.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2433.513800] LustreError: 78112:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2433.528647] LustreError: 78112:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2433.552445] LustreError: 78112:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2433.573328] LustreError: 78112:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2433.613600] Lustre: server umount lustre-MDT0000 complete [ 2433.622518] LustreError: 78112:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2436.391853] LustreError: 77552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2436.414675] LustreError: 77552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2440.676625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2440.836720] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2440.846809] LustreError: 75027:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2441.167295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.243661] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2441.512432] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2446.284821] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2446.324341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2446.361530] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2446.421328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2456.549297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.556629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2456.584789] Lustre: Skipped 2 previous similar messages [ 2456.603493] Lustre: Skipped 1 previous similar message [ 2460.962622] Lustre: server umount lustre-OST0000 complete [ 2466.793178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.798678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2466.808636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2466.831785] Lustre: Skipped 1 previous similar message [ 2470.832286] Lustre: server umount lustre-MDT0000 complete [ 2471.906772] LustreError: 77056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2471.920950] LustreError: 77056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2475.325231] LustreError: 75016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814302 with bad export cookie 9935438732014616721 [ 2475.350042] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2475.356092] LustreError: 75016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2475.790963] Lustre: server umount lustre-MDT0001 complete [ 2485.639096] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 2487.950470] Key type lgssc unregistered [ 2488.196699] LNet: 79907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.200886] LNetError: 79907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2488.215245] LNet: Removed LNI 192.168.204.117@tcp [ 2488.937209] Key type .llcrypt unregistered [ 2488.940339] Key type ._llcrypt unregistered [ 2499.633836] Key type ._llcrypt registered [ 2499.639260] Key type .llcrypt registered [ 2499.729043] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 2500.029441] Key type .llcrypt unregistered [ 2500.031599] Key type ._llcrypt unregistered [ 2521.442899] Key type ._llcrypt registered [ 2521.447351] Key type .llcrypt registered [ 2521.532994] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 2522.436947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2522.543717] alg: No test for adler32 (adler32-zlib) [ 2523.632447] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 2523.889564] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 2525.616198] Key type lgssc registered [ 2526.843751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2535.012320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2535.022060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.436901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2541.430883] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2546.666789] LustreError: 81816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.692173] LustreError: 81816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2550.764505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.413023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2556.625395] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2564.141132] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2569.034187] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2575.470757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.910143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.528632] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2582.073456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2587.935807] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2597.347521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2597.348390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.364742] LustreError: Skipped 1 previous similar message [ 2597.367785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2597.382624] Lustre: Skipped 1 previous similar message [ 2597.406092] Lustre: Skipped 1 previous similar message [ 2601.833809] Lustre: server umount lustre-OST0000 complete [ 2607.590448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2607.597246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.610061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2607.618424] Lustre: Skipped 1 previous similar message [ 2612.706683] LustreError: 81817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.745099] Lustre: server umount lustre-MDT0000 complete [ 2616.300794] LustreError: 81797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814443 with bad export cookie 10807743572812533185 [ 2616.310470] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2616.331593] LustreError: 81797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2616.747444] Lustre: server umount lustre-MDT0001 complete [ 2627.245985] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 2629.995872] Key type lgssc unregistered [ 2630.356192] LNet: 85351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2630.365448] LNetError: 85351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2630.378290] LNet: Removed LNI 192.168.204.117@tcp [ 2631.088073] Key type .llcrypt unregistered [ 2631.090138] Key type ._llcrypt unregistered [ 2658.616335] Key type ._llcrypt registered [ 2658.625767] Key type .llcrypt registered [ 2658.890797] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 2660.837936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2660.914302] alg: No test for adler32 (adler32-zlib) [ 2662.827973] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 2663.325656] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 2664.962551] Key type lgssc registered [ 2667.001322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2677.636328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2677.683634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.349552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.609704] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2689.527446] LustreError: 86764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.559798] LustreError: 86764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2694.637803] LustreError: 86763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2696.772055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2697.272598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2701.801394] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2708.779582] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2714.714394] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2721.155205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2721.495251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2724.618874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2729.895904] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2744.013408] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2753.507000] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2753.512740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.522224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2755.559811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2755.564177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2755.564206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.752537] Lustre: server umount lustre-OST0000 complete [ 2763.747978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2763.764417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.803234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.812901] Lustre: Skipped 1 previous similar message [ 2765.794807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.834545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.847796] Lustre: Skipped 1 previous similar message [ 2768.646918] Lustre: server umount lustre-MDT0000 complete [ 2770.917900] LustreError: 86763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.941517] LustreError: 86763:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2772.358529] LustreError: 86743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814599 with bad export cookie 11172910141195147572 [ 2772.365394] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2772.370920] LustreError: 86743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2772.662363] Lustre: server umount lustre-MDT0001 complete [ 2783.828877] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 2786.504222] Key type lgssc unregistered [ 2786.750875] LNet: 90302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2786.765615] LNetError: 90302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2786.782775] LNet: Removed LNI 192.168.204.117@tcp [ 2787.622473] Key type .llcrypt unregistered [ 2787.624386] Key type ._llcrypt unregistered [ 2816.707787] Key type ._llcrypt registered [ 2816.712079] Key type .llcrypt registered [ 2816.852956] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 2818.212511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2818.243652] alg: No test for adler32 (adler32-zlib) [ 2819.613705] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 2819.962289] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 2821.736228] Key type lgssc registered [ 2823.071897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2831.030702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2831.041661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.555354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2836.478375] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2843.109793] LustreError: 91722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2843.136728] LustreError: 91722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2845.905646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.733128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2852.020604] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2861.410310] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2870.634215] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2878.491268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2878.822195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.769069] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2894.369245] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2902.414303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2917.859376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2917.881934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2917.883978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2917.897996] LustreError: Skipped 1 previous similar message [ 2917.909370] Lustre: Skipped 1 previous similar message [ 2920.128890] Lustre: server umount lustre-OST0000 complete [ 2928.098479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.108149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2928.138952] Lustre: Skipped 2 previous similar messages [ 2928.159201] Lustre: Skipped 1 previous similar message [ 2930.817903] Lustre: server umount lustre-MDT0000 complete [ 2933.217119] LustreError: 91723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.244570] LustreError: 91723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2934.819692] LustreError: 91702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814761 with bad export cookie 12436050056523960737 [ 2934.820120] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2934.853611] LustreError: 91702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2935.589757] Lustre: server umount lustre-MDT0001 complete [ 2947.585513] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 2950.672720] Key type lgssc unregistered [ 2951.088226] LNet: 95282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2951.099920] LNetError: 95282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2951.116199] LNet: Removed LNI 192.168.204.117@tcp [ 2952.360618] Key type .llcrypt unregistered [ 2952.363464] Key type ._llcrypt unregistered [ 2978.788148] Key type ._llcrypt registered [ 2978.798295] Key type .llcrypt registered [ 2978.927288] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 2980.279139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2980.331225] alg: No test for adler32 (adler32-zlib) [ 2981.456618] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 2981.787348] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 2983.496545] Key type lgssc registered [ 2984.757569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2992.783841] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2992.809221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3008.353135] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3013.472753] LustreError: 96671:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.117@tcp: failed processing log, type 4: rc = -110 [ 3040.161849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.154383] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3057.976346] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3065.722431] Lustre: server umount lustre-OST0000 complete [ 3075.178755] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 13:28:18 (1786814898) [ 3083.896215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3084.443309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.531743] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3089.895762] LustreError: 98312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3089.919097] LustreError: 98312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3095.014348] LustreError: 98311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3099.925921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3100.130647] LustreError: 98312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.417902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3105.905717] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3114.608532] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3121.567115] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3130.393042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3130.852241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3137.908575] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3146.130295] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3154.422308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3166.691093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3166.694347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3166.712752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3169.763077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3169.766933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3169.778480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3172.538311] Lustre: server umount lustre-OST0000 complete [ 3177.442083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3177.460194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3177.493635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3177.509209] Lustre: Skipped 1 previous similar message [ 3180.003554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3180.006278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3185.157359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.167620] Lustre: Skipped 1 previous similar message [ 3191.265546] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3191.501774] Lustre: server umount lustre-MDT0000 complete [ 3194.340791] LustreError: 98312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3194.354858] LustreError: 98312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3196.476484] LustreError: 98293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815023 with bad export cookie 7099838736899353227 [ 3196.493218] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3196.503188] LustreError: 98293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3196.911633] Lustre: server umount lustre-MDT0001 complete [ 3210.177717] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 3213.338717] Key type lgssc unregistered [ 3213.617733] LNet: 102034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3213.629780] LNetError: 102034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3213.642455] LNet: Removed LNI 192.168.204.117@tcp [ 3214.561088] Key type .llcrypt unregistered [ 3214.563440] Key type ._llcrypt unregistered [ 3273.164388] Key type ._llcrypt registered [ 3273.167768] Key type .llcrypt registered [ 3273.428401] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 3277.613340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3277.775045] alg: No test for adler32 (adler32-zlib) [ 3279.719190] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 3280.526126] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 3282.601236] Key type lgssc registered [ 3284.664773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3298.816271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3298.874191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.209647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.321242] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3310.575106] LustreError: 104227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3310.620694] LustreError: 104227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3315.683678] LustreError: 104228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.618399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3317.104261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3322.304407] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3330.285996] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3336.748505] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3343.135748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3343.482953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3345.542733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3350.372540] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3358.631654] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3372.521053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3372.535955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3372.566066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.638150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.643934] Lustre: Skipped 1 previous similar message [ 3378.165747] Lustre: server umount lustre-OST0000 complete [ 3382.753346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.759451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.774122] Lustre: Skipped 2 previous similar messages [ 3382.791098] Lustre: Skipped 2 previous similar messages [ 3387.876092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.888146] Lustre: Skipped 1 previous similar message [ 3388.529315] Lustre: server umount lustre-MDT0000 complete [ 3393.002438] LustreError: 104228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.034409] LustreError: 104228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3393.196225] LustreError: 104207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815219 with bad export cookie 11609807464439350550 [ 3393.203804] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3393.208316] LustreError: 104207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3393.719884] Lustre: server umount lustre-MDT0001 complete [ 3404.184642] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 3406.946467] Key type lgssc unregistered [ 3407.228733] LNet: 107785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3407.237940] LNetError: 107785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3407.259246] LNet: Removed LNI 192.168.204.117@tcp [ 3408.348300] Key type .llcrypt unregistered [ 3408.353760] Key type ._llcrypt unregistered [ 3428.709849] Key type ._llcrypt registered [ 3428.712398] Key type .llcrypt registered [ 3428.800731] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 3429.983206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3430.029326] alg: No test for adler32 (adler32-zlib) [ 3431.077784] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 3431.314231] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 3432.971082] Key type lgssc registered [ 3434.028653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3443.899800] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3443.963127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3459.744740] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3464.865721] LustreError: 109028:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.117@tcp: failed processing log, type 4: rc = -110 [ 3491.618697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3497.130813] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3505.439925] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3513.315416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3513.644788] LustreError: MGC192.168.204.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3513.659834] LustreError: 110053:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3513.676497] LustreError: 110053:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3513.725197] LustreError: 110053:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3513.915082] Lustre: server umount lustre-MDT0000 complete [ 3513.919071] LustreError: 110053:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3534.304243] Lustre: 108365:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786815345/real 1786815345] req@000000008dff9af0 x1873611595975936/t0(0) o400->MGC192.168.204.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786815361 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3534.343649] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3534.369973] LustreError: 109040:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3554.908930] Lustre: server umount lustre-OST0000 complete [ 3565.083873] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 3577.996341] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 3619.817177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.028735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3620.047929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3620.112586] Lustre: lustre-MDT0000: new disk, initializing [ 3620.169329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3620.185995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3625.377913] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3641.502355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3641.685587] Lustre: 112852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3641.753220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3641.760151] Lustre: Skipped 1 previous similar message [ 3641.830152] Lustre: lustre-MDT0001: new disk, initializing [ 3641.988668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3642.059291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3642.090473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3647.559807] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3662.523326] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3669.511730] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3676.197845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3676.411478] Lustre: lustre-OST0000: new disk, initializing [ 3676.415651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3676.422671] Lustre: 114820:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3676.503975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3683.454548] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3686.492884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3686.504469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3686.723017] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3693.205759] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3699.394475] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3699.736598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3703.877693] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3704.121901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3707.361111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3707.369596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3707.374560] LustreError: Skipped 1 previous similar message [ 3707.378845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3707.392975] Lustre: Skipped 1 previous similar message [ 3707.406212] Lustre: Skipped 1 previous similar message [ 3712.483304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3712.486853] Lustre: Skipped 1 previous similar message [ 3717.610239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3717.621179] Lustre: Skipped 1 previous similar message [ 3720.160885] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3720.423348] Lustre: server umount lustre-OST0000 complete [ 3727.842680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3727.857573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3727.869356] Lustre: Skipped 1 previous similar message [ 3732.961551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3732.968996] Lustre: Skipped 2 previous similar messages [ 3739.107900] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3739.231533] Lustre: server umount lustre-MDT0000 complete [ 3742.376361] LustreError: 112843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815569 with bad export cookie 16063900138355634777 [ 3742.382534] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3742.398155] LustreError: 112843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3742.712452] Lustre: server umount lustre-MDT0001 complete [ 3753.100943] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 13:39:36 (1786815576) [ 3757.367315] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3776.439897] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 3787.152779] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 3820.617910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.871110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3820.888213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3820.987996] Lustre: lustre-MDT0000: new disk, initializing [ 3821.128108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3821.167537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3825.554503] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3836.945937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.054817] Lustre: 119724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3837.116648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3837.124448] Lustre: Skipped 1 previous similar message [ 3837.184762] Lustre: lustre-MDT0001: new disk, initializing [ 3837.249673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3837.283597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3837.294344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3841.584737] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3852.236947] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3857.604844] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3865.140776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3865.367994] Lustre: lustre-OST0000: new disk, initializing [ 3865.371825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3865.378036] Lustre: 121688:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3865.466781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3871.341735] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3874.362193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3874.374067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3874.494461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3880.438761] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3885.126948] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3885.282389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3889.873535] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3890.026389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3895.275358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3895.279643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.292437] LustreError: Skipped 1 previous similar message [ 3895.293768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3895.293774] Lustre: Skipped 2 previous similar messages [ 3895.344949] Lustre: Skipped 1 previous similar message [ 3906.016234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3906.176979] Lustre: server umount lustre-OST0000 complete [ 3914.209256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3914.224494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.240899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3914.254818] Lustre: Skipped 5 previous similar messages [ 3919.454472] Lustre: server umount lustre-MDT0000 complete [ 3923.329545] LustreError: 119717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815750 with bad export cookie 16063900138355635652 [ 3923.331089] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3923.339216] LustreError: 119717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3923.597497] Lustre: server umount lustre-MDT0001 complete [ 3931.515179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3931.783391] LustreError: 123908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3931.806137] LustreError: 123908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3931.850093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3936.294956] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3937.251130] LustreError: 123909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3941.369446] LustreError: 123908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3944.157846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3948.774586] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3958.050240] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3964.423811] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3972.969640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3980.107350] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3989.273691] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3995.949823] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4000.227171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.235335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4000.249504] Lustre: Skipped 1 previous similar message [ 4000.251919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4000.272633] Lustre: Skipped 3 previous similar messages [ 4002.404244] Lustre: server umount lustre-OST0000 complete [ 4010.477150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.510034] Lustre: Skipped 2 previous similar messages [ 4012.761609] Lustre: server umount lustre-MDT0000 complete [ 4015.585297] LustreError: 124636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4015.599221] LustreError: 124636:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4016.624702] LustreError: 123888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815843 with bad export cookie 16063900138355636541 [ 4016.637758] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4016.642286] LustreError: 123888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4017.213407] Lustre: server umount lustre-MDT0001 complete [ 4027.317805] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 4029.781420] Key type lgssc unregistered [ 4030.158888] LNet: 127550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4030.183960] LNetError: 127550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4030.198075] LNet: Removed LNI 192.168.204.117@tcp [ 4030.981132] Key type .llcrypt unregistered [ 4030.983499] Key type ._llcrypt unregistered [ 4049.724643] Key type ._llcrypt registered [ 4049.727228] Key type .llcrypt registered [ 4049.830195] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 4067.727806] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 4069.612794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4069.651128] alg: No test for adler32 (adler32-zlib) [ 4070.975314] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 4071.415927] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 4073.144870] Key type lgssc registered [ 4074.239556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4113.023804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4113.061631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4114.374796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4114.438122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4114.526933] Lustre: lustre-MDT0000: new disk, initializing [ 4114.608967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4114.633275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4119.752596] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4131.740629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4131.989327] Lustre: 130709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4132.066409] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4132.075096] Lustre: Skipped 1 previous similar message [ 4132.237878] Lustre: lustre-MDT0001: new disk, initializing [ 4132.419301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4132.472507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4132.482694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4138.198372] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4148.651740] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4154.784724] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4162.398586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4162.862612] Lustre: lustre-OST0000: new disk, initializing [ 4162.867658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4162.874196] Lustre: 132679:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4162.988471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4169.903779] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4171.393698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4171.420466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4171.509638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4180.043660] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4186.812711] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4187.164455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4193.626256] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4193.944866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4197.353823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4197.359666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.412982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4197.428559] Lustre: Skipped 1 previous similar message [ 4202.466337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4202.474776] Lustre: Skipped 1 previous similar message [ 4202.677319] Lustre: server umount lustre-OST0000 complete [ 4212.707576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4212.709797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4212.729478] Lustre: Skipped 2 previous similar messages [ 4216.265643] Lustre: server umount lustre-MDT0000 complete [ 4220.277660] LustreError: 130700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816047 with bad export cookie 12799541065840956093 [ 4220.281146] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4220.294583] LustreError: 130700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4221.000619] Lustre: server umount lustre-MDT0001 complete [ 4230.441411] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 13:47:34 (1786816054) [ 4236.956577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4237.444946] LustreError: 135092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.479383] LustreError: 135092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4237.555878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.071965] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4242.919957] LustreError: 135093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4248.035956] LustreError: 135092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4251.237370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4251.689485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4255.324687] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4263.477036] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4269.355533] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4272.100555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4272.113754] LustreError: Skipped 1 previous similar message [ 4272.121159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4272.131453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4272.136345] Lustre: Skipped 1 previous similar message [ 4277.221984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4277.229920] Lustre: Skipped 2 previous similar messages [ 4278.038519] Lustre: server umount lustre-MDT0000 complete [ 4282.303442] LustreError: 136663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816109 with bad export cookie 12799541065840956975 [ 4282.313126] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4282.318446] LustreError: 136663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4282.816178] Lustre: server umount lustre-MDT0001 complete [ 4292.330624] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 13:48:36 (1786816116) [ 4299.026264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4314.594931] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4319.714329] LustreError: 137508:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.117@tcp: failed processing log, type 4: rc = -110 [ 4345.442410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4354.043742] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4363.374912] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4366.242753] Lustre: server umount lustre-OST0000 complete [ 4376.833099] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 13:50:00 (1786816200) [ 4384.721360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4385.342297] LustreError: 138956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4385.448579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4390.367761] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4390.885570] LustreError: 138957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.496721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4405.738925] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4414.177058] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4421.935951] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4429.080922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4429.325805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.333535] Lustre: Skipped 1 previous similar message [ 4437.010499] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4447.569619] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4455.906245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4455.915663] Lustre: Skipped 1 previous similar message [ 4455.919381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4461.352640] Lustre: server umount lustre-MDT0000 complete [ 4466.146843] LustreError: 138957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4466.166731] LustreError: 138957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4466.797260] LustreError: 138936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816293 with bad export cookie 12799541065840957633 [ 4466.814889] LustreError: 138936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4466.816405] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4467.331960] Lustre: server umount lustre-MDT0001 complete [ 4482.232455] Lustre: server umount lustre-OST0000 complete [ 4493.226984] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 13:51:56 (1786816316) [ 4500.386035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.896417] LustreError: 142613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.988315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4506.166670] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4517.821317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4523.351854] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4532.166808] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4540.176944] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4552.037691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4562.300926] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4570.393207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4572.355813] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4578.623329] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4578.865669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4583.349720] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4583.634482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4585.954330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4585.955489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4585.975219] Lustre: Skipped 2 previous similar messages [ 4585.976666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4585.976672] Lustre: Skipped 5 previous similar messages [ 4591.973469] Lustre: server umount lustre-OST0000 complete [ 4596.195725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4596.213084] Lustre: Skipped 2 previous similar messages [ 4602.269527] Lustre: server umount lustre-MDT0000 complete [ 4606.441121] LustreError: 142614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4606.476908] LustreError: 142614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4606.614806] LustreError: 142593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816433 with bad export cookie 12799541065840958711 [ 4606.627244] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4606.629601] LustreError: 142593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4607.097891] Lustre: server umount lustre-MDT0001 complete [ 4617.026322] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 13:54:01 (1786816441) [ 4625.257170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4640.864668] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4645.984642] LustreError: 146775:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.117@tcp: failed processing log, type 4: rc = -110 [ 4671.776841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.789865] Lustre: Skipped 2 previous similar messages [ 4678.358805] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4685.694237] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4692.480940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4693.003665] LustreError: 146800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4693.175331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4697.746935] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4705.896095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4711.172371] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4720.413307] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4727.071667] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4733.087311] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4733.315795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4737.795416] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4737.993923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4742.114173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4742.121914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4742.152504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4742.157840] Lustre: Skipped 7 previous similar messages [ 4746.518762] Lustre: server umount lustre-MDT0000 complete [ 4750.115733] LustreError: 148554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816576 with bad export cookie 12799541065840959733 [ 4750.117189] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4750.129353] LustreError: 148554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4750.573721] Lustre: server umount lustre-MDT0001 complete [ 4765.072961] Lustre: server umount lustre-OST0000 complete [ 4774.576762] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 13:56:38 (1786816598) [ 4781.985741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4797.664799] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4802.786216] LustreError: 150972:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.117@tcp: failed processing log, type 4: rc = -110 [ 4834.151174] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4841.372748] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4848.939454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4849.473561] LustreError: 150997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4849.499841] LustreError: 150997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4849.689772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4855.229892] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4865.676943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4870.807847] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4878.394791] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4883.788259] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4889.556226] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4889.773953] Lustre: lustre-OST0001: new disk, initializing [ 4889.781292] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4889.790968] Lustre: 153842:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4895.518563] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4898.376350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4898.395485] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4898.642222] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4905.410675] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4909.915896] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4910.252627] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4916.198386] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4916.602636] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4921.319072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4921.345987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4921.385337] Lustre: Skipped 2 previous similar messages [ 4921.410189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4921.428587] Lustre: Skipped 3 previous similar messages [ 4925.763131] Lustre: server umount lustre-OST0000 complete [ 4934.636239] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4934.653028] LustreError: Skipped 1 previous similar message [ 4936.098552] Lustre: server umount lustre-OST0001 complete [ 4946.683614] Lustre: server umount lustre-MDT0000 complete [ 4950.621182] LustreError: 150978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816777 with bad export cookie 12799541065840960685 [ 4950.633060] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4950.639572] LustreError: 150978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4950.932152] Lustre: server umount lustre-MDT0001 complete [ 4977.521831] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 14:00:02 (1786816802) [ 4979.934205] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4981.813976] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4984.031397] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 14:00:07 (1786816807) [ 4991.108129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4991.426362] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4991.454687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4991.697448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4991.714771] Lustre: Skipped 6 previous similar messages [ 4995.611749] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5003.385028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.521941] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5003.562059] Lustre: 157402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5007.763667] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5014.725090] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5019.896824] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5025.396417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5025.538978] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5030.827714] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5032.570912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5037.351942] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5041.543444] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5041.708447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5045.663411] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5045.880885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5047.779262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5047.785154] LustreError: Skipped 2 previous similar messages [ 5047.788237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5047.804125] Lustre: Skipped 5 previous similar messages [ 5053.412344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5053.418136] Lustre: Skipped 12 previous similar messages [ 5053.748982] Lustre: server umount lustre-OST0000 complete [ 5074.616536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5078.950271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.963632] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5080.813875] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5084.084622] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.119204] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5088.270947] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5093.803996] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5094.063531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5099.057403] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5099.267661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5105.243166] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e2b464c3000.ost_server_uuid 50 [ 5107.138691] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8e2b464c3000.ost_server_uuid in FULL state after 0 sec [ 5107.364324] LustreError: 157410:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.17@tcp) returned error from blocking AST (req@00000000b9cb9123 x1873612267331712 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000058c0e532/0xb1a11ac8d579ae07 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.17@tcp remote: 0x43dabaf2b9d07855 expref: 5 pid: 158806 timeout: 5207 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5107.445485] LustreError: lustre-MDT0000: A client on nid 192.168.204.17@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5107.463197] LustreError: 157399:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.17@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000058c0e532/0xb1a11ac8d579ae07 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.17@tcp remote: 0x43dabaf2b9d07855 expref: 6 pid: 158806 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5112.807733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5117.285729] Lustre: server umount lustre-OST0000 complete [ 5136.352128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5136.642408] Lustre: server umount lustre-MDT0000 complete [ 5139.431329] LustreError: 157409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5139.451579] LustreError: 157409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 5140.031227] LustreError: 157394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816966 with bad export cookie 12799541065840961861 [ 5140.037607] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5140.043848] LustreError: 157394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5146.577838] Lustre: server umount lustre-MDT0001 complete [ 5157.731870] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 5160.180484] Key type lgssc unregistered [ 5160.505490] LNet: 163534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5160.513325] LNetError: 163534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5160.535388] LNet: Removed LNI 192.168.204.117@tcp [ 5161.435728] Key type .llcrypt unregistered [ 5161.439491] Key type ._llcrypt unregistered [ 5183.431685] Key type ._llcrypt registered [ 5183.435349] Key type .llcrypt registered [ 5183.587595] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 5184.952623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5185.037348] alg: No test for adler32 (adler32-zlib) [ 5186.034715] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 5186.243571] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 5187.912280] Key type lgssc registered [ 5189.103243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5197.131790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5197.145113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5198.677314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5203.684937] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5209.063378] LustreError: 164942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5209.100917] LustreError: 164942:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5212.625737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5212.901191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5217.231963] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5227.320638] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5235.328749] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5244.782787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5245.148631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5252.094331] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5260.765594] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5268.741150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5277.181189] Lustre: Failing over lustre-MDT0000 [ 5277.672276] Lustre: server umount lustre-MDT0000 complete [ 5279.201853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5279.206754] LustreError: 164938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5279.223711] Lustre: Skipped 2 previous similar messages [ 5279.250964] LustreError: 164938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5284.323675] LustreError: 164943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5284.360821] LustreError: 164943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5289.444135] LustreError: 164938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5289.459101] LustreError: 164938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5289.939242] LustreError: 164922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.17@tcp arrived at 1786817116 with bad export cookie 12567467688740441542 [ 5289.957982] LustreError: 164922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5294.562468] Lustre: 164251:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786817105/real 1786817105] req@00000000a64dcb38 x1873613435207040/t0(0) o400->MGC192.168.204.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786817121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5294.564486] LustreError: 165671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5294.583707] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5294.596882] LustreError: 165671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5297.645625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.813269] LustreError: 164250:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005d0de7bd x1873613435211904/t0(0) o250->MGC192.168.204.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5304.844474] LustreError: 165671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5304.865439] LustreError: 165671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5305.267800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5305.390300] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5309.821363] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5310.467604] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5310.505464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5312.208910] Lustre: lustre-MDT0000: Denying connection for new client d576dae7-1a29-4b57-a49b-569509805816 (at 192.168.204.17@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5320.673858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5320.681872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5320.685867] Lustre: Skipped 1 previous similar message [ 5320.704242] Lustre: Skipped 1 previous similar message [ 5325.799905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5325.807465] Lustre: Skipped 1 previous similar message [ 5326.620281] Lustre: server umount lustre-OST0000 complete [ 5330.888679] LustreError: 168698:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5330.900507] Lustre: 168047:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5330.910857] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5330.921539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5330.936682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5330.939126] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5330.950630] LustreError: 168047:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f489e708 x1873613435227392/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5330.980045] LustreError: 168047:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5330.992348] LustreError: 168047:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5331.000970] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5331.323749] Lustre: server umount lustre-MDT0000 complete [ 5334.611108] LustreError: 164923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817161 with bad export cookie 12567467688740441871 [ 5334.617328] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5334.621242] LustreError: 164250:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5334.623470] LustreError: 164923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5341.008742] Lustre: server umount lustre-MDT0001 complete [ 5349.940969] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 5352.295806] Key type lgssc unregistered [ 5352.576445] LNet: 169453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5352.582297] LNetError: 169453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5352.605317] LNet: Removed LNI 192.168.204.117@tcp [ 5353.461050] Key type .llcrypt unregistered [ 5353.462784] Key type ._llcrypt unregistered [ 5374.323954] Key type ._llcrypt registered [ 5374.326768] Key type .llcrypt registered [ 5374.473737] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 5375.506651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5375.642528] alg: No test for adler32 (adler32-zlib) [ 5376.718543] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 5376.960066] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 5378.649795] Key type lgssc registered [ 5379.823828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5387.167793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5387.183222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.590747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5392.152781] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5399.021146] LustreError: 170847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5399.054882] LustreError: 170847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5399.824154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5400.160721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5404.132855] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5410.886856] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5416.582529] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5422.828235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5423.211956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5428.723777] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5433.459874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5435.397150] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5444.065665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5444.072415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.085481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5444.108842] Lustre: Skipped 1 previous similar message [ 5449.038121] Lustre: server umount lustre-OST0000 complete [ 5454.307900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.325311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5454.335643] Lustre: Skipped 1 previous similar message [ 5454.341453] Lustre: Skipped 1 previous similar message [ 5458.559320] Lustre: server umount lustre-MDT0000 complete [ 5459.426372] LustreError: 170842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.442169] LustreError: 170842:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5462.630679] LustreError: 170827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817289 with bad export cookie 14111427510791828633 [ 5462.634454] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5462.645170] LustreError: 170827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5462.965417] Lustre: server umount lustre-MDT0001 complete [ 5473.165775] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 5476.158826] Key type lgssc unregistered [ 5476.450534] LNet: 174382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5476.459751] LNetError: 174382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5476.472625] LNet: Removed LNI 192.168.204.117@tcp [ 5477.424853] Key type .llcrypt unregistered [ 5477.429472] Key type ._llcrypt unregistered [ 5507.944860] Key type ._llcrypt registered [ 5507.948565] Key type .llcrypt registered [ 5508.049638] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 5509.169724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5509.207278] alg: No test for adler32 (adler32-zlib) [ 5510.202144] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 5510.387257] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 5512.040863] Key type lgssc registered [ 5513.351903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5520.563358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5520.582230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.089813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5526.531942] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5531.625143] LustreError: 176085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5531.643281] LustreError: 176085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5534.673665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.117445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5539.039559] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5546.411270] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5551.963442] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5558.453393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5558.730212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5564.508400] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5567.022401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5571.986022] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5580.257402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5580.266333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.281425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5582.308893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5582.315961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5582.321059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5584.431190] Lustre: server umount lustre-OST0000 complete [ 5591.523496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5591.534468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.552371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5591.562877] Lustre: Skipped 1 previous similar message [ 5593.881712] Lustre: server umount lustre-MDT0000 complete [ 5597.666433] LustreError: 176065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817424 with bad export cookie 192822169961987243 [ 5597.669439] LustreError: 176085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5597.683771] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5597.966154] Lustre: server umount lustre-MDT0001 complete [ 5608.734459] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 5611.640163] Key type lgssc unregistered [ 5611.938220] LNet: 179622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5611.949385] LNetError: 179622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5611.979854] LNet: Removed LNI 192.168.204.117@tcp [ 5612.952872] Key type .llcrypt unregistered [ 5612.958473] Key type ._llcrypt unregistered [ 5637.056289] Key type ._llcrypt registered [ 5637.060890] Key type .llcrypt registered [ 5637.218630] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 5638.341928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5638.370909] alg: No test for adler32 (adler32-zlib) [ 5639.440977] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 5639.640843] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 5641.305932] Key type lgssc registered [ 5642.823592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5653.487408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5653.509088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5654.812120] Lustre: *** cfs_fail_loc=135, val=0*** [ 5654.898654] LustreError: 181145:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5654.903253] LustreError: 181145:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.117@tcp: cfg command failed: rc = -2 [ 5654.908226] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5654.908226] [ 5654.914498] LustreError: MGC192.168.204.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5654.921180] LustreError: 181106:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5654.926595] LustreError: 181106:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5654.930304] LustreError: 181106:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5654.937177] LustreError: 181106:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5655.009945] Lustre: server umount lustre-MDT0000 complete [ 5655.014324] LustreError: 181106:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5660.458624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.717499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5664.682311] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5670.882567] LustreError: 181518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5670.894565] LustreError: 181518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5671.788784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5672.179856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5675.876784] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5682.929429] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5688.191396] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5690.854559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5690.874822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5690.882334] Lustre: Skipped 1 previous similar message [ 5695.972634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5696.286790] Lustre: server umount lustre-MDT0000 complete [ 5697.003544] LustreError: 181518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5697.028219] LustreError: 181518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5699.846178] LustreError: 181498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817526 with bad export cookie 16916197595863964119 [ 5699.857764] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5699.860938] LustreError: 181498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5700.148716] Lustre: server umount lustre-MDT0001 complete [ 5710.298267] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 5712.696597] Key type lgssc unregistered [ 5712.932234] LNet: 183842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5712.941787] LNetError: 183842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5712.967783] LNet: Removed LNI 192.168.204.117@tcp [ 5713.872666] Key type .llcrypt unregistered [ 5713.878050] Key type ._llcrypt unregistered [ 5734.662850] Key type ._llcrypt registered [ 5734.665177] Key type .llcrypt registered [ 5734.765085] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 5735.215667] Key type .llcrypt unregistered [ 5735.218483] Key type ._llcrypt unregistered [ 5748.729174] Key type ._llcrypt registered [ 5748.732833] Key type .llcrypt registered [ 5748.822296] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 5749.560832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5749.655052] alg: No test for adler32 (adler32-zlib) [ 5750.833207] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 5751.116307] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 5752.801074] Key type lgssc registered [ 5754.062761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5761.703429] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5761.736247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5777.254789] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5782.369114] LustreError: 185958:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.117@tcp: failed processing log, type 4: rc = -110 [ 5809.057163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5815.972230] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5822.747067] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5829.936703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.340732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.405957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5835.551079] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5835.746460] LustreError: 185983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5835.780086] LustreError: 185983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5840.870433] LustreError: 185984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.946596] LustreError: 185983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.984200] LustreError: 185983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5845.171887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.645792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5850.451674] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5858.320605] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5865.587528] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5868.317773] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5881.313936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5881.321991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5881.333873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5881.831830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5881.848776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5881.874503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5885.360905] Lustre: server umount lustre-OST0000 complete [ 5891.554680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5891.558689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5891.572626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5897.186077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5897.196751] Lustre: Skipped 3 previous similar messages [ 5901.282984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5901.290811] Lustre: Skipped 1 previous similar message [ 5904.352204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5904.575605] Lustre: server umount lustre-MDT0000 complete [ 5906.407483] LustreError: 187000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5906.445280] LustreError: 187000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5908.588620] LustreError: 185965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817735 with bad export cookie 701355595350504674 [ 5908.590878] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5908.614755] LustreError: 185965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5909.005251] Lustre: server umount lustre-MDT0001 complete [ 5918.774226] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 5921.367653] Key type lgssc unregistered [ 5921.717821] LNet: 189930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5921.723250] LNetError: 189930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5921.735295] LNet: Removed LNI 192.168.204.117@tcp [ 5922.451774] Key type .llcrypt unregistered [ 5922.454212] Key type ._llcrypt unregistered [ 5942.120987] Key type ._llcrypt registered [ 5942.123670] Key type .llcrypt registered [ 5942.229690] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 5943.289845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5943.366277] alg: No test for adler32 (adler32-zlib) [ 5944.487452] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 5944.725490] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 5946.441276] Key type lgssc registered [ 5947.548934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5954.734858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5954.767958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5956.199953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5959.967940] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5965.286328] LustreError: 191327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5965.312912] LustreError: 191327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5967.824141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5968.238833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5972.163464] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 5979.327092] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5984.954292] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5994.339133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5994.994085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6001.205202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6004.042556] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6017.276693] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6025.219036] Lustre: Failing over lustre-MDT0000 [ 6025.486554] Lustre: server umount lustre-MDT0000 complete [ 6025.590208] LustreError: 191323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6027.753039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6027.761864] LustreError: 191324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6027.783879] Lustre: Skipped 2 previous similar messages [ 6027.819115] LustreError: 191324:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6030.625082] LustreError: 191322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.744411] LustreError: 192656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.760219] LustreError: 192656:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6044.134924] Lustre: 190640:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786817854/real 1786817854] req@00000000e4e2457b x1873614231074944/t0(0) o400->MGC192.168.204.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786817870 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6044.185077] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6044.203450] LustreError: 191327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6044.244916] LustreError: 191327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6045.328294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6053.356194] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xaeddfda196e42f35 [ 6053.377399] Lustre: MGC192.168.204.117@tcp: Connection restored to 0@lo (at 0@lo) [ 6053.620726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6053.681568] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6057.255781] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6058.022721] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6058.993852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6059.011027] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6059.075725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6062.558672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6068.767591] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6074.346830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6074.351440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6074.360679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6074.385135] Lustre: Skipped 1 previous similar message [ 6074.406536] Lustre: Skipped 1 previous similar message [ 6076.176082] Lustre: server umount lustre-OST0000 complete [ 6084.588827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6084.591159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6084.625635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6084.633128] Lustre: Skipped 1 previous similar message [ 6086.400324] Lustre: server umount lustre-MDT0000 complete [ 6089.699827] LustreError: 191327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6089.718166] LustreError: 191327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6089.878489] LustreError: 191309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817916 with bad export cookie 12600506202915614517 [ 6089.883272] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6089.896581] LustreError: 191309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6090.354532] Lustre: server umount lustre-MDT0001 complete [ 6100.525027] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 6102.834694] Key type lgssc unregistered [ 6103.072384] LNet: 196170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6103.080455] LNetError: 196170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6103.102397] LNet: Removed LNI 192.168.204.117@tcp [ 6103.940218] Key type .llcrypt unregistered [ 6103.944741] Key type ._llcrypt unregistered [ 6124.985706] Key type ._llcrypt registered [ 6124.987633] Key type .llcrypt registered [ 6125.072227] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 6126.124703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6126.158204] alg: No test for adler32 (adler32-zlib) [ 6127.187696] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 6127.452568] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 6129.112243] Key type lgssc registered [ 6130.093072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6137.626784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6137.659662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.174256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6143.388512] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6148.583826] LustreError: 197565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6148.594251] LustreError: 197565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6151.466213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6151.757735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6156.095790] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6169.110638] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6179.022133] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6186.911528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6187.437693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6194.231628] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6203.429679] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6208.946092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6210.931611] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6211.850426] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6225.548150] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6233.578289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6233.590948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6233.625607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6234.595784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6234.603701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6234.614673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6234.630507] Lustre: Skipped 1 previous similar message [ 6239.045289] Lustre: server umount lustre-OST0000 complete [ 6243.809697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6243.816415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6243.828275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6249.946140] Lustre: server umount lustre-MDT0000 complete [ 6249.955637] LustreError: 199672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6249.979640] LustreError: 199672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6254.771792] LustreError: 198299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818081 with bad export cookie 5688068161027385442 [ 6254.772758] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6254.784521] LustreError: 198299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6255.161282] Lustre: server umount lustre-MDT0001 complete [ 6267.947343] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 6270.765721] Key type lgssc unregistered [ 6271.020885] LNet: 201419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6271.027801] LNetError: 201419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6271.038957] LNet: Removed LNI 192.168.204.117@tcp [ 6271.899128] Key type .llcrypt unregistered [ 6271.902042] Key type ._llcrypt unregistered [ 6289.037567] Key type ._llcrypt registered [ 6289.043560] Key type .llcrypt registered [ 6289.153613] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 6290.353173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6290.425479] alg: No test for adler32 (adler32-zlib) [ 6291.479712] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 6291.753728] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 6293.472234] Key type lgssc registered [ 6294.685282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6302.570941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6302.586341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6304.112258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6309.167450] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6313.454061] LustreError: 202682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6313.491990] LustreError: 202682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6318.209917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6318.581764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6323.131677] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6331.629611] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6337.868149] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6345.888242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6346.223181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6348.297414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6355.353576] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6365.668401] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6371.576772] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6385.267276] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6392.958715] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6399.832846] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6414.823835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6414.824715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.834031] LustreError: Skipped 1 previous similar message [ 6414.835325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6414.844123] Lustre: Skipped 1 previous similar message [ 6414.855787] Lustre: Skipped 1 previous similar message [ 6416.829618] Lustre: server umount lustre-OST0000 complete [ 6425.084812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6425.088342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6425.101211] Lustre: Skipped 1 previous similar message [ 6425.118528] Lustre: Skipped 1 previous similar message [ 6427.538957] Lustre: server umount lustre-MDT0000 complete [ 6430.192237] LustreError: 202676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6430.220417] LustreError: 202676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6430.911571] LustreError: 202664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818257 with bad export cookie 1578106890763103151 [ 6430.921702] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6430.925098] LustreError: 202664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6431.397643] Lustre: server umount lustre-MDT0001 complete [ 6442.638379] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 6445.854892] Key type lgssc unregistered [ 6446.089732] LNet: 207709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6446.098630] LNetError: 207709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6446.123700] LNet: Removed LNI 192.168.204.117@tcp [ 6447.043716] Key type .llcrypt unregistered [ 6447.046770] Key type ._llcrypt unregistered [ 6469.115334] Key type ._llcrypt registered [ 6469.117538] Key type .llcrypt registered [ 6469.267944] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 6470.319075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6470.426447] alg: No test for adler32 (adler32-zlib) [ 6471.601767] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 6471.814163] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 6473.442764] Key type lgssc registered [ 6474.538859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6482.762399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6482.784139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6484.184925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6488.346694] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6493.672750] LustreError: 209102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6493.699803] LustreError: 209102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6495.999881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.300476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6500.745299] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6507.609507] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6511.686141] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6517.067496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6517.316628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6522.134233] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6528.643209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6530.091836] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6542.310465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6542.322146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6542.337610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6544.354684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6544.355474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6547.527323] Lustre: server umount lustre-OST0000 complete [ 6552.546651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6552.554490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6552.575172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6552.578483] Lustre: Skipped 1 previous similar message [ 6554.610391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6554.613331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6557.805218] Lustre: server umount lustre-MDT0000 complete [ 6559.714890] LustreError: 209831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6561.429678] LustreError: 209081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818388 with bad export cookie 16058384369597176641 [ 6561.447305] LustreError: 209081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6561.447440] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6561.750153] Lustre: server umount lustre-MDT0001 complete [ 6572.546929] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 6575.141964] Key type lgssc unregistered [ 6575.457743] LNet: 212791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6575.467223] LNetError: 212791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6575.488905] LNet: Removed LNI 192.168.204.117@tcp [ 6576.294710] Key type .llcrypt unregistered [ 6576.297071] Key type ._llcrypt unregistered [ 6593.871983] Key type ._llcrypt registered [ 6593.874559] Key type .llcrypt registered [ 6594.021837] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 6594.928558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6594.954282] alg: No test for adler32 (adler32-zlib) [ 6596.029569] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 6596.302527] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 6597.968331] Key type lgssc registered [ 6599.180767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6609.323039] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6609.342936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6610.745386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6616.668156] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6621.157230] LustreError: 214049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6621.173554] LustreError: 214049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6626.294520] LustreError: 214050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6626.814832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.322488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6632.961522] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6642.351753] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6650.185634] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6660.579050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.397526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6669.994945] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6679.533896] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6680.877100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6696.421150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6696.425539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6696.440250] Lustre: Skipped 1 previous similar message [ 6696.448473] Lustre: Skipped 1 previous similar message [ 6701.537564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6701.547282] Lustre: Skipped 1 previous similar message [ 6701.888970] Lustre: server umount lustre-OST0000 complete [ 6706.658126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6706.659893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6706.693799] Lustre: Skipped 1 previous similar message [ 6706.708516] Lustre: Skipped 1 previous similar message [ 6711.784219] LustreError: 214779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6711.803961] LustreError: 214779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6711.866973] Lustre: server umount lustre-MDT0000 complete [ 6715.666623] LustreError: 215897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818542 with bad export cookie 16953656859740181467 [ 6715.677267] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6715.690966] LustreError: 215897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6716.105656] Lustre: server umount lustre-MDT0001 complete [ 6726.618905] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 6729.766082] Key type lgssc unregistered [ 6730.099671] LNet: 217738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6730.107687] LNetError: 217738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6730.129908] LNet: Removed LNI 192.168.204.117@tcp [ 6731.144291] Key type .llcrypt unregistered [ 6731.146141] Key type ._llcrypt unregistered [ 6749.308212] Key type ._llcrypt registered [ 6749.315597] Key type .llcrypt registered [ 6749.471510] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 6750.673865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6750.729341] alg: No test for adler32 (adler32-zlib) [ 6751.812734] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 6752.129757] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 6753.904190] Key type lgssc registered [ 6755.204883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6763.686805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6763.722404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6765.469477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6770.277235] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6775.798224] LustreError: 218995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6775.821799] LustreError: 218995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6779.096992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6779.447368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6783.850182] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6792.921764] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6798.932294] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6806.309428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6806.677394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6810.749276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6813.669893] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6821.710358] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6833.822184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6833.906518] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6834.162903] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6842.805387] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 6852.395563] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6854.053458] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6866.582653] Lustre: Permanently deactivating lustre-OST0001 [ 6866.616451] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6873.723227] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6880.326605] Lustre: Permanently reactivating lustre-OST0001 [ 6880.338504] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6880.344136] Lustre: Skipped 2 previous similar messages [ 6883.380217] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6883.400373] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6883.411450] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6883.424476] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6883.484785] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6892.980528] Lustre: server umount lustre-OST0001 complete [ 6894.048953] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6894.053783] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6894.060641] LustreError: Skipped 1 previous similar message [ 6894.071796] LustreError: 220817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6894.103124] Lustre: Skipped 2 previous similar messages [ 6894.130922] LustreError: 220817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6898.145545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6898.159637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6898.188705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6899.698312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6899.698791] LustreError: 220818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6899.731738] Lustre: Skipped 1 previous similar message [ 6899.795485] LustreError: 220818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6903.734222] Lustre: server umount lustre-OST0000 complete [ 6909.922970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6909.925757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6909.944656] Lustre: Skipped 2 previous similar messages [ 6909.961377] Lustre: Skipped 1 previous similar message [ 6913.826755] Lustre: server umount lustre-MDT0000 complete [ 6915.042571] LustreError: 218995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6915.061807] LustreError: 218995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6918.461645] LustreError: 218974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818745 with bad export cookie 1663436282619368675 [ 6918.464523] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6918.487663] LustreError: 218974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6918.862951] Lustre: server umount lustre-MDT0001 complete [ 6930.446653] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 6933.557352] Key type lgssc unregistered [ 6933.850601] LNet: 224045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6933.864089] LNetError: 224045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6933.894115] LNet: Removed LNI 192.168.204.117@tcp [ 6934.750915] Key type .llcrypt unregistered [ 6934.752771] Key type ._llcrypt unregistered [ 6973.403962] Key type ._llcrypt registered [ 6973.406992] Key type .llcrypt registered [ 6973.509687] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 6974.652661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6974.713481] alg: No test for adler32 (adler32-zlib) [ 6975.759774] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 6975.974632] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 6977.648693] Key type lgssc registered [ 6978.749217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6986.652095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6986.671184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6987.859421] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6987.886222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6988.058940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6992.730773] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7002.817687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7002.997317] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7003.051942] Lustre: 226254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7003.362696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7007.603943] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7014.930741] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7021.123985] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7028.433783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7028.649241] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7028.889914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7035.258678] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7036.146098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7042.929422] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7046.690622] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7052.390692] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7061.560726] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7070.696692] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7085.631593] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7085.645340] Lustre: Skipped 1 previous similar message [ 7104.217453] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7104.222625] Lustre: Skipped 2 previous similar messages [ 7144.432681] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7144.438422] Lustre: Skipped 4 previous similar messages [ 7208.970405] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7208.976442] Lustre: Skipped 7 previous similar messages [ 7215.592474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7215.598636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7215.604642] LustreError: Skipped 1 previous similar message [ 7215.618163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7215.619579] Lustre: Skipped 1 previous similar message [ 7215.646040] Lustre: Skipped 1 previous similar message [ 7220.713416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7220.718594] Lustre: Skipped 1 previous similar message [ 7221.236155] Lustre: server umount lustre-OST0000 complete [ 7225.832599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7225.838428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7225.843224] Lustre: Skipped 1 previous similar message [ 7225.854881] Lustre: Skipped 1 previous similar message [ 7230.948551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7230.960051] Lustre: Skipped 1 previous similar message [ 7236.065988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7236.075710] Lustre: Skipped 1 previous similar message [ 7239.136301] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7239.399965] Lustre: server umount lustre-MDT0000 complete [ 7243.175570] LustreError: 229266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819069 with bad export cookie 15051036810828251385 [ 7243.182884] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7243.183664] LustreError: 229266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7243.807916] Lustre: server umount lustre-MDT0001 complete [ 7254.338904] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 7257.170728] Key type lgssc unregistered [ 7257.416761] LNet: 231065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7257.422601] LNetError: 231065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7257.431638] LNet: Removed LNI 192.168.204.117@tcp [ 7258.374275] Key type .llcrypt unregistered [ 7258.377747] Key type ._llcrypt unregistered [ 7282.833121] Key type ._llcrypt registered [ 7282.836309] Key type .llcrypt registered [ 7282.955746] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 7283.986993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7284.022429] alg: No test for adler32 (adler32-zlib) [ 7284.995538] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 7285.248710] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 7286.920356] Key type lgssc registered [ 7288.268244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7295.903885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7295.927050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7297.374651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7300.949257] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7307.752964] LustreError: 232480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7307.768517] LustreError: 232480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7308.373991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7308.684856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7312.645097] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7319.881697] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7326.034849] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7332.569045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7332.913562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7338.034439] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7343.159230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7344.356487] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7359.461612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7359.468836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7359.480690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7361.538346] Lustre: server umount lustre-OST0000 complete [ 7365.089675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7365.096770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7365.105738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7368.675967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7368.677119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7373.798789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7373.808698] Lustre: Skipped 2 previous similar messages [ 7375.006297] Lustre: server umount lustre-MDT0000 complete [ 7378.916944] LustreError: 233208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7378.940692] LustreError: 233208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7379.745595] LustreError: 233215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819206 with bad export cookie 5570704462807728769 [ 7379.748424] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7379.759914] LustreError: 233215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7380.367232] Lustre: server umount lustre-MDT0001 complete [ 7392.039127] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 7395.231587] Key type lgssc unregistered [ 7395.610350] LNet: 236173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7395.620117] LNetError: 236173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7395.642480] LNet: Removed LNI 192.168.204.117@tcp [ 7396.475574] Key type .llcrypt unregistered [ 7396.480788] Key type ._llcrypt unregistered [ 7420.798409] Key type ._llcrypt registered [ 7420.801291] Key type .llcrypt registered [ 7420.929665] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 7421.871401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7421.940204] alg: No test for adler32 (adler32-zlib) [ 7423.065130] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 7423.253613] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 7424.888168] Key type lgssc registered [ 7426.148752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7698.111978] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 7700.814475] Key type lgssc unregistered [ 7701.125533] LNet: 238202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7701.138351] LNetError: 238202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7701.155815] LNet: Removed LNI 192.168.204.117@tcp [ 7702.097494] Key type .llcrypt unregistered [ 7702.101773] Key type ._llcrypt unregistered [ 7723.315773] Key type ._llcrypt registered [ 7723.318653] Key type .llcrypt registered [ 7723.423874] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 7736.753263] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 7737.554069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7737.657523] alg: No test for adler32 (adler32-zlib) [ 7738.727986] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 7738.984900] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 7740.680479] Key type lgssc registered [ 7741.701194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7768.154199] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7498 sec ========== 14:46:32 (1786819592) [ 7769.940519] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:46:34 (1786819594) === [ 7773.890260] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:46:37 (1786819597) === [ 7809.357793] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 7811.986097] Key type lgssc unregistered [ 7812.247544] LNet: 243446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7812.256949] LNetError: 243446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7812.287012] LNet: Removed LNI 192.168.204.117@tcp [ 7813.033843] Key type .llcrypt unregistered [ 7813.036477] Key type ._llcrypt unregistered