[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 493511637 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003233] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007917] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010139] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.012051] SELinux: Initializing. [ 0.014036] *** VALIDATE selinux *** [ 0.022424] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027487] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028233] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030122] *** VALIDATE tmpfs *** [ 0.031431] *** VALIDATE proc *** [ 0.033271] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.036257] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037173] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040027] Spectre V2 : User space: Vulnerable [ 0.041015] Speculative Store Bypass: Vulnerable [ 0.044294] debug: unmapping init [mem 0xffffffff98e59000-0xffffffff98e60fff] [ 0.046209] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047761] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048024] ... version: 2 [ 0.049014] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052014] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.053993] ... event mask: 000000070000000f [ 0.054321] rcu: Hierarchical SRCU implementation. [ 0.056485] smp: Bringing up secondary CPUs ... [ 0.057657] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.061558] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150056] node 0 deferred pages initialised in 83ms [ 0.152098] devtmpfs: initialized [ 0.153273] x86/mm: Memory block size: 128MB [ 0.155795] gcov: version magic: 0x41383552 [ 0.159320] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160107] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161283] pinctrl core: initialized pinctrl subsystem [ 0.162211] [ 0.162831] ************************************************************* [ 0.163014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164018] ** ** [ 0.165019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166021] ** ** [ 0.167017] ** This means that this kernel is built to expose internal ** [ 0.168015] ** IOMMU data structures, which may compromise security on ** [ 0.169019] ** your system. ** [ 0.170012] ** ** [ 0.171010] ** If you see this message and you are not debugging the ** [ 0.172011] ** kernel, report this immediately to your vendor! ** [ 0.173010] ** ** [ 0.174006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175011] ************************************************************* [ 0.176589] NET: Registered protocol family 16 [ 0.177371] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.178060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.179065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180459] cpuidle: using governor menu [ 0.182800] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185480] PCI: Using configuration type 1 for base access [ 0.187140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.197112] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.199109] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.203088] cryptd: max_cpu_qlen set to 1000 [ 0.205289] ACPI: Added _OSI(Module Device) [ 0.206014] ACPI: Added _OSI(Processor Device) [ 0.207015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.208015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.212045] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.214705] ACPI: Interpreter enabled [ 0.215079] ACPI: PM: (supports S0 S3 S4 S5) [ 0.216014] ACPI: Using IOAPIC for interrupt routing [ 0.217103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.218712] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.227818] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.228047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.229020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.230083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.232435] acpiphp: Slot [2] registered [ 0.233129] acpiphp: Slot [5] registered [ 0.234127] acpiphp: Slot [6] registered [ 0.235148] acpiphp: Slot [7] registered [ 0.236154] acpiphp: Slot [8] registered [ 0.237157] acpiphp: Slot [9] registered [ 0.238130] acpiphp: Slot [10] registered [ 0.239183] acpiphp: Slot [3] registered [ 0.240116] acpiphp: Slot [4] registered [ 0.241154] acpiphp: Slot [11] registered [ 0.243152] acpiphp: Slot [12] registered [ 0.245117] acpiphp: Slot [13] registered [ 0.246104] acpiphp: Slot [14] registered [ 0.248103] acpiphp: Slot [15] registered [ 0.249122] acpiphp: Slot [16] registered [ 0.251101] acpiphp: Slot [17] registered [ 0.252091] acpiphp: Slot [18] registered [ 0.254125] acpiphp: Slot [19] registered [ 0.255134] acpiphp: Slot [20] registered [ 0.257095] acpiphp: Slot [21] registered [ 0.258122] acpiphp: Slot [22] registered [ 0.260146] acpiphp: Slot [23] registered [ 0.261099] acpiphp: Slot [24] registered [ 0.263099] acpiphp: Slot [25] registered [ 0.264098] acpiphp: Slot [26] registered [ 0.266091] acpiphp: Slot [27] registered [ 0.267113] acpiphp: Slot [28] registered [ 0.268093] acpiphp: Slot [29] registered [ 0.270123] acpiphp: Slot [30] registered [ 0.271089] acpiphp: Slot [31] registered [ 0.272097] PCI host bridge to bus 0000:00 [ 0.274022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.276027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.279033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.281031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.283019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.285024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.286186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.289007] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.292729] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.301488] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.307557] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.309025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.312021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.314014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.316960] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.319643] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.322055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.324714] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.330014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.342014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.346021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.352038] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.359015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.366016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.387020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.400708] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.407015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.413027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.432023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.440972] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.450026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.458019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.478017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.490506] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.496019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.503016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.520019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.530590] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.539016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.547018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.558014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.568110] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.572014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.579028] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.601030] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.613065] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.615243] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.617450] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.619398] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.621217] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.627122] iommu: Default domain type: Passthrough [ 0.629528] SCSI subsystem initialized [ 0.630111] ACPI: bus type USB registered [ 0.631091] usbcore: registered new interface driver usbfs [ 0.633108] usbcore: registered new interface driver hub [ 0.634052] usbcore: registered new device driver usb [ 0.635098] pps_core: LinuxPPS API ver. 1 registered [ 0.637014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.639054] PTP clock support registered [ 0.641078] EDAC MC: Ver: 3.0.0 [ 0.642532] PCI: Using ACPI for IRQ routing [ 0.644136] NetLabel: Initializing [ 0.645011] NetLabel: domain hash size = 128 [ 0.646019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.647144] NetLabel: unlabeled traffic allowed by default [ 0.649100] vgaarb: loaded [ 0.650286] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.651025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.655252] clocksource: Switched to clocksource kvm-clock [ 0.761968] VFS: Disk quotas dquot_6.6.0 [ 0.763820] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.766567] *** VALIDATE ramfs *** [ 0.768014] *** VALIDATE hugetlbfs *** [ 0.769911] pnp: PnP ACPI init [ 0.772692] pnp: PnP ACPI: found 6 devices [ 0.792838] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.796529] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.799073] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.801508] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.804123] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.806754] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.809559] NET: Registered protocol family 2 [ 0.812649] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.817917] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.821373] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.826631] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.829916] TCP: Hash tables configured (established 65536 bind 65536) [ 0.832399] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.835415] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.837941] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.840758] NET: Registered protocol family 1 [ 0.843616] RPC: Registered named UNIX socket transport module. [ 0.845675] RPC: Registered udp transport module. [ 0.847192] RPC: Registered tcp transport module. [ 0.848806] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.851086] NET: Registered protocol family 44 [ 0.852938] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.855027] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.857534] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.859393] PCI: CLS 0 bytes, default 64 [ 0.860763] Unpacking initramfs... [ 2.227318] debug: unmapping init [mem 0xffff8c547cc54000-0xffff8c547ffbffff] [ 2.231135] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.232535] software IO TLB: mapped [mem 0x00000000a3200000-0x00000000a7200000] (64MB) [ 2.234131] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.687819] Initialise system trusted keyrings [ 2.688941] Key type blacklist registered [ 2.690284] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.696648] zbud: loaded [ 2.698875] *** VALIDATE nfs *** [ 2.699780] *** VALIDATE nfs4 *** [ 2.700876] pstore: using deflate compression [ 2.703273] Platform Keyring initialized [ 2.802910] NET: Registered protocol family 38 [ 2.805677] Key type asymmetric registered [ 2.808471] Asymmetric key parser 'x509' registered [ 2.810493] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.813596] io scheduler mq-deadline registered [ 2.815702] io scheduler kyber registered [ 2.817469] io scheduler bfq registered [ 2.819220] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.821775] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.824720] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.827394] ACPI: Power Button [PWRF] [ 2.832687] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.839911] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.856833] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.874928] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.894042] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.922200] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.952206] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.957651] Non-volatile memory driver v1.3 [ 2.958791] Linux agpgart interface v0.103 [ 2.985315] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.988508] vda: detected capacity change from 0 to 75100160 [ 3.017512] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.020628] vdb: detected capacity change from 0 to 1073741824 [ 3.038545] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.040900] vdc: detected capacity change from 0 to 2621440000 [ 3.057657] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.061127] vdd: detected capacity change from 0 to 2621440000 [ 3.082336] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.086558] vde: detected capacity change from 0 to 4294967296 [ 3.105945] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.109809] vdf: detected capacity change from 0 to 4294967296 [ 3.120692] libphy: Fixed MDIO Bus: probed [ 3.127839] usbcore: registered new interface driver usbserial_generic [ 3.130642] usbserial: USB Serial support registered for generic [ 3.132953] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.137721] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.138919] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.140697] mousedev: PS/2 mouse device common for all mice [ 3.144289] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.144728] rtc_cmos 00:05: RTC can wake from S4 [ 3.153780] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.153951] rtc_cmos 00:05: registered as rtc0 [ 3.159292] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.162962] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.165850] intel_pstate: CPU model not supported [ 3.169344] hid: raw HID events driver (C) Jiri Kosina [ 3.171522] usbcore: registered new interface driver usbhid [ 3.173586] usbhid: USB HID core driver [ 3.175229] drop_monitor: Initializing network drop monitor service [ 3.177912] Initializing XFRM netlink socket [ 3.180165] NET: Registered protocol family 10 [ 3.183172] Segment Routing with IPv6 [ 3.184782] NET: Registered protocol family 17 [ 3.187643] mpls_gso: MPLS GSO support [ 3.192677] RAS: Correctable Errors collector initialized. [ 3.195513] AVX version of gcm_enc/dec engaged. [ 3.197675] AES CTR mode by8 optimization enabled [ 3.283797] sched_clock: Marking stable (3283763717, 0)->(4214350977, -930587260) [ 3.287610] registered taskstats version 1 [ 3.289624] Loading compiled-in X.509 certificates [ 3.292325] zswap: loaded using pool lzo/zbud [ 3.324185] Key type big_key registered [ 3.339851] Key type encrypted registered [ 3.341425] ima: No TPM chip found, activating TPM-bypass! [ 3.343285] ima: Allocated hash algorithm: sha1 [ 3.344829] ima: No architecture policies found [ 3.346825] evm: Initialising EVM extended attributes: [ 3.348844] evm: security.selinux [ 3.350067] evm: security.ima [ 3.351150] evm: security.capability [ 3.352470] evm: HMAC attrs: 0x1 [ 3.355190] rtc_cmos 00:05: setting system clock to 2026-09-08 01:57:28 UTC (1788832648) [ 3.361653] debug: unmapping init [mem 0xffffffff99e03000-0xffffffff99ffffff] [ 3.364881] debug: unmapping init [mem 0xffffffff98b82000-0xffffffff98e58fff] [ 3.374077] Write protecting the kernel read-only data: 28672k [ 3.377752] debug: unmapping init [mem 0xffffffff97203000-0xffffffff973fffff] [ 3.380505] debug: unmapping init [mem 0xffffffff97b14000-0xffffffff97bfffff] [ 3.416167] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.425646] systemd[1]: Detected virtualization kvm. [ 3.427594] systemd[1]: Detected architecture x86-64. [ 3.429952] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.460477] systemd[1]: No hostname configured. [ 3.462303] systemd[1]: Set hostname to . [ 3.464537] random: systemd: uninitialized urandom read (16 bytes read) [ 3.467075] systemd[1]: Initializing machine ID from random generator. [ 3.524337] random: ln: uninitialized urandom read (6 bytes read) [ 3.593835] random: systemd: uninitialized urandom read (16 bytes read) [ 3.597256] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.601914] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.610120] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.187574] device-mapper: uevent: version 1.0.3 [ 4.191158] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK [ 4.841017] random: fast init done ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.894617] virtio_net virtio0 ens2: renamed from eth0 [ 4.975438] scsi host0: ata_piix [ 4.990677] scsi host1: ata_piix [ 4.992214] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.994662] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.839206] dracut-initqueue[575]: RTNETLINK answers: File exists [ 9.860721] random: crng init done [ 9.862347] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.349908] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.599739] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.856460] SELinux: Disabled at runtime. [ 11.918506] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.924393] systemd[1]: Detected virtualization kvm. [ 11.925518] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.391926] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.396116] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.401522] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.406380] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.409432] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.417159] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.424109] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK [0[ 12.558219] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.946372] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.306950] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.362257] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.547458] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.564986] EDAC sbridge: Ver: 1.1.2 [ 15.443726] Key type dns_resolver registered [ 15.762595] NFS: Registering the id_resolver key type [ 15.765820] Key type id_resolver registered [ 15.767503] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg125-server login: [ 42.528492] libcfs: loading out-of-tree module taints kernel. [ 42.547313] Key type ._llcrypt registered [ 42.549026] Key type .llcrypt registered [ 42.605538] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 51.477273] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 52.267494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.278132] alg: No test for adler32 (adler32-zlib) [ 53.357432] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 53.700253] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 55.327225] Key type lgssc registered [ 56.903893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.842149] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 120.703037] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 135.883300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 135.922079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 137.410552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 137.455575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 137.684963] Lustre: lustre-MDT0000: new disk, initializing [ 137.819829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 137.839407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 142.156867] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.412162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.573365] Lustre: 6481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 158.643935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 158.647399] Lustre: Skipped 1 previous similar message [ 158.853237] Lustre: lustre-MDT0001: new disk, initializing [ 158.973318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.048677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.069325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 163.015036] hrtimer: interrupt took 7102860 ns [ 164.001303] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.677574] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 183.382485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 183.549274] Lustre: lustre-OST0000: new disk, initializing [ 183.550749] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 183.553802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 183.560864] Lustre: 8420:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 183.603399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.936303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 188.941406] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 188.965528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 191.172606] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.389576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.669417] Lustre: lustre-OST0001: new disk, initializing [ 209.676707] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.691989] Lustre: 9490:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.810115] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.934302] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.719564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 218.750118] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 218.899439] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 230.632746] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 241.554283] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 248.859933] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing check_logdir /tmp/testlogs/ [ 253.010748] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing yml_node [ 256.803305] Lustre: DEBUG MARKER: Client: 2.17.57.106 [ 259.345924] Lustre: DEBUG MARKER: MDS: 2.17.57.106 [ 261.942778] Lustre: DEBUG MARKER: OSS: 2.17.57.106 [ 263.383758] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 22:01:47 EDT 2026 [ 281.484995] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 283.115549] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 301.032515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.039223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.063740] Lustre: Skipped 3 previous similar messages [ 301.081468] Lustre: Skipped 3 previous similar messages [ 306.154765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.167072] Lustre: Skipped 3 previous similar messages [ 311.285698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.299302] Lustre: Skipped 3 previous similar messages [ 313.311887] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 315.571709] Lustre: server umount lustre-MDT0000 complete [ 320.209284] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832965 with bad export cookie 9242804949440464125 [ 320.212837] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.245422] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 321.505822] LustreError: 6486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.506367] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.506533] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.520159] LustreError: 6486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 321.551293] Lustre: Skipped 1 previous similar message [ 325.538959] LustreError: 6488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.551898] LustreError: 6488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 327.060340] Lustre: server umount lustre-MDT0001 complete [ 331.433510] Lustre: server umount lustre-OST0000 complete [ 335.748982] Lustre: server umount lustre-OST0001 complete [ 344.045851] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 355.204769] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 399.049860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.294945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 399.330253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 399.395702] Lustre: lustre-MDT0000: new disk, initializing [ 399.591701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.611170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 404.747031] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 416.040056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.192284] Lustre: 15062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 416.236708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 416.240356] Lustre: Skipped 1 previous similar message [ 416.293857] Lustre: lustre-MDT0001: new disk, initializing [ 416.388542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 416.451791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 416.467803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 422.210698] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 433.643335] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 440.227669] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 448.069589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 448.463957] Lustre: lustre-OST0000: new disk, initializing [ 448.468257] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 448.472327] Lustre: 17030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 448.563240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.979812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.995418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 450.160232] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 455.884379] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 466.713526] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 472.060486] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 472.246927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 478.151077] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 478.419412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 480.736251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.740595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.743557] LustreError: Skipped 1 previous similar message [ 480.744656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.744662] Lustre: Skipped 3 previous similar messages [ 480.793583] Lustre: Skipped 1 previous similar message [ 495.071489] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 495.340896] Lustre: server umount lustre-OST0000 complete [ 501.217160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.230245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 501.232551] Lustre: Skipped 5 previous similar messages [ 506.083134] Lustre: server umount lustre-MDT0000 complete [ 506.348756] LustreError: 15074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.364388] LustreError: 15074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 510.397428] LustreError: 15055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833155 with bad export cookie 9242804949440465665 [ 510.415802] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 510.967951] Lustre: server umount lustre-MDT0001 complete [ 515.177853] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:05:59 (1788833159) [ 522.600736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.276343] LustreError: 19298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.352126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.891836] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 528.354792] LustreError: 19299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.766992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.955287] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 555.004811] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 561.746176] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 570.105647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 578.477749] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 589.862955] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 603.622836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.623112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 603.626280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 603.635505] Lustre: Skipped 2 previous similar messages [ 609.045671] Lustre: server umount lustre-OST0000 complete [ 613.867123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 613.890186] Lustre: Skipped 1 previous similar message [ 619.613484] Lustre: server umount lustre-MDT0000 complete [ 623.882550] LustreError: 19279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833269 with bad export cookie 9242804949440466547 [ 623.895108] LustreError: 19279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 623.897570] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 624.385358] Lustre: server umount lustre-MDT0001 complete [ 639.045299] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 641.950879] Key type lgssc unregistered [ 642.293525] LNet: 22854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.299447] LNetError: 22854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 642.317134] LNet: Removed LNI 192.168.201.125@tcp [ 643.213147] Key type .llcrypt unregistered [ 643.217267] Key type ._llcrypt unregistered [ 669.239384] Key type ._llcrypt registered [ 669.242256] Key type .llcrypt registered [ 669.387965] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 670.349689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 670.379151] alg: No test for adler32 (adler32-zlib) [ 671.568819] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 671.977206] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 673.642135] Key type lgssc registered [ 674.830803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 684.123309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 684.159310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 685.677296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 691.377285] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 695.268127] LustreError: 24263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 695.311091] LustreError: 24263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 700.391587] LustreError: 24264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 704.045945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.974842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 710.461407] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 722.031976] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 729.677409] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 737.953540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 738.460675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 746.745426] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 755.315545] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 755.888086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 775.656584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 775.668791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.684338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 776.672288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 776.676805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 776.689879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 776.724923] Lustre: Skipped 1 previous similar message [ 779.207693] Lustre: server umount lustre-OST0000 complete [ 786.915042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.930726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 786.942760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 786.979758] Lustre: Skipped 1 previous similar message [ 790.517043] Lustre: server umount lustre-MDT0000 complete [ 792.035180] LustreError: 24264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 792.063689] LustreError: 24264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 795.115065] LustreError: 24243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833440 with bad export cookie 15594092354829060026 [ 795.127829] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 795.133066] LustreError: 24243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 795.662427] Lustre: server umount lustre-MDT0001 complete [ 810.439326] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 814.367532] Key type lgssc unregistered [ 814.818091] LNet: 28243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.826807] LNetError: 28243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.864775] LNet: Removed LNI 192.168.201.125@tcp [ 816.191090] Key type .llcrypt unregistered [ 816.196891] Key type ._llcrypt unregistered [ 850.485024] Key type ._llcrypt registered [ 850.488671] Key type .llcrypt registered [ 850.625615] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 852.683919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 852.744517] alg: No test for adler32 (adler32-zlib) [ 853.867382] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 854.213692] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 856.008943] Key type lgssc registered [ 857.712366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 867.322443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 867.356062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.184721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 874.621505] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 879.589909] LustreError: 29652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.620820] LustreError: 29652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 884.706511] LustreError: 29653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 886.733969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.362379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 892.291820] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 901.594565] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 909.144265] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 926.371845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 926.935199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 934.406441] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 939.179942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 942.412130] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 950.762471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 950.773726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 950.797728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.825795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 953.840216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.855336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 954.871695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 954.886729] Lustre: Skipped 1 previous similar message [ 956.873411] Lustre: server umount lustre-OST0000 complete [ 964.064940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 964.074485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.089560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 970.210585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 970.219539] Lustre: Skipped 3 previous similar messages [ 975.327610] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 975.525841] Lustre: server umount lustre-MDT0000 complete [ 980.454297] LustreError: 29652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 980.481929] LustreError: 29652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 980.795760] LustreError: 29634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833625 with bad export cookie 723220015659942553 [ 980.811353] LustreError: 29634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 980.812400] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 981.390751] Lustre: server umount lustre-MDT0001 complete [ 995.043175] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 998.710694] Key type lgssc unregistered [ 999.246788] LNet: 33538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.252092] LNetError: 33538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.293091] LNet: Removed LNI 192.168.201.125@tcp [ 1000.871447] Key type .llcrypt unregistered [ 1000.874505] Key type ._llcrypt unregistered [ 1029.485582] Key type ._llcrypt registered [ 1029.487642] Key type .llcrypt registered [ 1029.684940] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1031.790358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1031.843250] alg: No test for adler32 (adler32-zlib) [ 1032.924668] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 1033.236780] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1034.976042] Key type lgssc registered [ 1036.605188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1047.104568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1047.175294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.948193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1054.485380] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1059.309643] LustreError: 34949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1059.339697] LustreError: 34949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1064.422922] LustreError: 34948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.698888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.238551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1070.572354] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1080.700457] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1088.712462] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1097.974438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1098.402963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1106.235591] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1114.707309] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1118.920890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1129.440760] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1129.447383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.462332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1129.479493] Lustre: Skipped 1 previous similar message [ 1129.501780] Lustre: Skipped 1 previous similar message [ 1134.579353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1134.598591] Lustre: Skipped 1 previous similar message [ 1134.824122] Lustre: server umount lustre-OST0000 complete [ 1139.699123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.710276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.721125] Lustre: Skipped 1 previous similar message [ 1144.801077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.821372] Lustre: Skipped 2 previous similar messages [ 1145.163233] Lustre: server umount lustre-MDT0000 complete [ 1149.928474] LustreError: 36095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.967987] LustreError: 36095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1150.158230] LustreError: 34929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833795 with bad export cookie 3427982003636700872 [ 1150.200154] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1151.002482] Lustre: server umount lustre-MDT0001 complete [ 1169.779182] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1174.494493] Key type lgssc unregistered [ 1174.877943] LNet: 38528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1174.895539] LNetError: 38528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1174.936823] LNet: Removed LNI 192.168.201.125@tcp [ 1176.126460] Key type .llcrypt unregistered [ 1176.128770] Key type ._llcrypt unregistered [ 1210.739882] Key type ._llcrypt registered [ 1210.745539] Key type .llcrypt registered [ 1210.909413] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1212.709468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1212.803407] alg: No test for adler32 (adler32-zlib) [ 1213.867759] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 1214.220985] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1216.007414] Key type lgssc registered [ 1217.283246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1226.549816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1226.562051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.241603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1234.736108] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1238.499083] LustreError: 39940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1238.531831] LustreError: 39940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1243.630607] LustreError: 39939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.740264] LustreError: 39940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1249.278375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.826717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1256.429871] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1267.287751] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1275.287401] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1283.879529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.315696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.031694] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1298.679410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1300.446533] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1308.815313] Lustre: lustre-OST0000: Not available for connect from 192.168.201.25@tcp (stopping) [ 1309.165942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.165942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.175431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1313.906669] Lustre: lustre-OST0000: Not available for connect from 192.168.201.25@tcp (stopping) [ 1313.914598] Lustre: Skipped 2 previous similar messages [ 1314.298543] Lustre: server umount lustre-OST0000 complete [ 1336.801989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1336.810671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.830315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1339.886374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1339.897289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.572876] Lustre: server umount lustre-MDT0000 complete [ 1344.999453] LustreError: 39936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.016307] LustreError: 39936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1345.724741] LustreError: 39922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833990 with bad export cookie 11973744347378521644 [ 1345.743298] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1345.746654] LustreError: 39922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1360.351191] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1360.798358] Lustre: server umount lustre-MDT0001 complete [ 1376.004282] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1379.298991] Key type lgssc unregistered [ 1379.581956] LNet: 43650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.596076] LNetError: 43650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.620827] LNet: Removed LNI 192.168.201.125@tcp [ 1380.620434] Key type .llcrypt unregistered [ 1380.623760] Key type ._llcrypt unregistered [ 1412.230714] Key type ._llcrypt registered [ 1412.238592] Key type .llcrypt registered [ 1412.421402] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1414.537778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1414.572338] alg: No test for adler32 (adler32-zlib) [ 1415.898842] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 1416.229082] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1417.960345] Key type lgssc registered [ 1419.879254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1428.164231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1428.187351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1429.639660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.245254] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1439.215934] LustreError: 45059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1439.239864] LustreError: 45059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1444.328984] LustreError: 45058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1446.483678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1447.088103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1452.871202] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1462.592390] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1470.660300] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1478.963339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1479.425395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1486.207223] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1494.641305] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1499.900221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1505.250604] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1505.254788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1505.276155] Lustre: Skipped 1 previous similar message [ 1505.303391] Lustre: Skipped 2 previous similar messages [ 1505.936359] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.25@tcp (stopping) [ 1507.207836] Lustre: server umount lustre-MDT0000 complete [ 1510.373449] LustreError: 47552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1510.394330] LustreError: 47552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1511.712982] LustreError: 45039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834156 with bad export cookie 11705549913277128328 [ 1511.737106] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1515.494298] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.495645] LustreError: 45053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1515.507878] Lustre: Skipped 1 previous similar message [ 1515.528926] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1515.534903] LustreError: 45053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1521.264578] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.25@tcp (stopping) [ 1521.265271] LustreError: 46113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1521.282460] Lustre: Skipped 2 previous similar messages [ 1521.324775] LustreError: 46113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1526.239182] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1526.396916] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.25@tcp (stopping) [ 1526.417021] Lustre: Skipped 1 previous similar message [ 1526.841520] Lustre: server umount lustre-MDT0001 complete [ 1543.943776] Lustre: server umount lustre-OST0000 complete [ 1568.141653] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1571.518252] Key type lgssc unregistered [ 1571.780793] LNet: 48924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.786112] LNetError: 48924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.800796] LNet: Removed LNI 192.168.201.125@tcp [ 1572.655913] Key type .llcrypt unregistered [ 1572.662102] Key type ._llcrypt unregistered [ 1601.670262] Key type ._llcrypt registered [ 1601.673608] Key type .llcrypt registered [ 1601.937191] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1603.285743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1603.313612] alg: No test for adler32 (adler32-zlib) [ 1604.760189] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 1605.053185] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1606.767558] Key type lgssc registered [ 1608.039454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1615.976393] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1616.017045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1631.711579] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1636.831352] LustreError: 50306:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 1663.524340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1672.749913] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1682.417216] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1848.398556] Lustre: server umount lustre-OST0000 complete [ 1866.079485] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1868.805839] Key type lgssc unregistered [ 1869.157286] LNet: 51989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.164195] LNetError: 51989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.190538] LNet: Removed LNI 192.168.201.125@tcp [ 1869.940391] Key type .llcrypt unregistered [ 1869.942918] Key type ._llcrypt unregistered [ 1896.950394] Key type ._llcrypt registered [ 1896.952923] Key type .llcrypt registered [ 1897.097746] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1898.162656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1898.179431] alg: No test for adler32 (adler32-zlib) [ 1899.253620] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 1899.511649] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1901.208499] Key type lgssc registered [ 1902.328941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1911.686939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1911.720459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1913.554735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.563567] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1923.059510] LustreError: 53404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1923.077057] LustreError: 53404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1928.168349] LustreError: 53405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1929.915442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.467372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1935.366446] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1943.515165] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1950.502281] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1958.407596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1958.976065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1965.248807] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1972.715432] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1980.384931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1980.399332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.424585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1981.724423] Lustre: server umount lustre-OST0000 complete [ 1987.046050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1987.054067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.073277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1988.578860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1988.593567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.697163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1993.708335] Lustre: Skipped 2 previous similar messages [ 1998.817854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1998.824546] Lustre: Skipped 1 previous similar message [ 2000.351302] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2000.593365] Lustre: server umount lustre-MDT0000 complete [ 2003.939308] LustreError: 53404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.958959] LustreError: 53404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2004.184883] LustreError: 53384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834649 with bad export cookie 2484497912756834324 [ 2004.193861] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.764236] Lustre: server umount lustre-MDT0001 complete [ 2017.529355] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2020.253507] Key type lgssc unregistered [ 2020.472239] LNet: 56933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.477849] LNetError: 56933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2021.541844] LNet: Removed LNI 192.168.201.125@tcp [ 2022.194121] Key type .llcrypt unregistered [ 2022.195879] Key type ._llcrypt unregistered [ 2046.634240] Key type ._llcrypt registered [ 2046.636924] Key type .llcrypt registered [ 2046.834437] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2048.056455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2048.107403] alg: No test for adler32 (adler32-zlib) [ 2049.232535] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2049.586137] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2051.396406] Key type lgssc registered [ 2052.689394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2061.482688] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2061.510514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2077.087562] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2082.207346] LustreError: 58319:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 2108.959965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.360890] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2125.356770] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2134.186930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.762681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2134.845035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2139.832228] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2140.144396] LustreError: 58344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2140.165731] LustreError: 58344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2143.219596] LustreError: 59354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2143.239937] LustreError: 59354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2148.323695] LustreError: 59355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.363543] LustreError: 59355:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2149.516101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.848290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2156.327827] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2164.989752] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2171.999246] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2175.467600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2175.480354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2175.490410] Lustre: Skipped 1 previous similar message [ 2175.508833] Lustre: Skipped 1 previous similar message [ 2180.586679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2180.595697] Lustre: Skipped 1 previous similar message [ 2180.767469] Lustre: server umount lustre-OST0000 complete [ 2204.965044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2204.972484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2204.995343] Lustre: Skipped 1 previous similar message [ 2205.031876] Lustre: Skipped 1 previous similar message [ 2210.277351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2210.421995] Lustre: server umount lustre-MDT0000 complete [ 2214.324663] LustreError: 58325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834859 with bad export cookie 4009027109696214954 [ 2214.339402] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2220.780647] Lustre: server umount lustre-MDT0001 complete [ 2234.562619] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2237.199818] Key type lgssc unregistered [ 2237.457648] LNet: 62063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2237.466930] LNetError: 62063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2237.484811] LNet: Removed LNI 192.168.201.125@tcp [ 2238.415383] Key type .llcrypt unregistered [ 2238.416737] Key type ._llcrypt unregistered [ 2264.925225] Key type ._llcrypt registered [ 2264.927563] Key type .llcrypt registered [ 2265.032359] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2265.882265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2266.056158] alg: No test for adler32 (adler32-zlib) [ 2267.103786] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2267.333913] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2268.951665] Key type lgssc registered [ 2270.424140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2278.789630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2278.806941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.191207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2285.356631] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2290.665325] LustreError: 63484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2290.698063] LustreError: 63484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2294.046256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.484462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2299.088799] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2307.763613] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2313.483949] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2321.451261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2321.979217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2325.055856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2330.374849] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2341.424885] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2360.298249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.298817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2360.321516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2360.339661] Lustre: Skipped 1 previous similar message [ 2365.415450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2365.421287] Lustre: Skipped 1 previous similar message [ 2366.253490] Lustre: server umount lustre-OST0000 complete [ 2371.553281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2371.568644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.580335] Lustre: Skipped 1 previous similar message [ 2371.588415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2375.651544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2375.654221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2375.687054] Lustre: Skipped 1 previous similar message [ 2377.298631] Lustre: server umount lustre-MDT0000 complete [ 2380.769211] LustreError: 63479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.803959] LustreError: 63479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2382.717872] LustreError: 64219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835027 with bad export cookie 17771672485439379561 [ 2382.747937] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.295403] Lustre: server umount lustre-MDT0001 complete [ 2399.712801] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2402.665117] Key type lgssc unregistered [ 2403.117932] LNet: 67037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2403.121908] LNetError: 67037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2403.146700] LNet: Removed LNI 192.168.201.125@tcp [ 2404.219625] Key type .llcrypt unregistered [ 2404.226356] Key type ._llcrypt unregistered [ 2432.725796] Key type ._llcrypt registered [ 2432.732319] Key type .llcrypt registered [ 2432.997055] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2434.221071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2434.283717] alg: No test for adler32 (adler32-zlib) [ 2435.655780] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2436.027708] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2437.791337] Key type lgssc registered [ 2439.000417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2449.943584] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 22:38:14 (1788835094) [ 2458.675296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2458.720931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.428295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.177905] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2469.871373] LustreError: 68743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.893602] LustreError: 68743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2474.982790] LustreError: 68742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.381778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.944940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2482.430247] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2490.504588] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2497.173522] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2504.627189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2505.123943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.167199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2511.212338] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2519.497387] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2527.984995] Lustre: Failing over lustre-MDT0000 [ 2528.226879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2528.243776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2528.260192] LustreError: 68742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.282767] Lustre: server umount lustre-MDT0000 complete [ 2531.512363] LustreError: 68738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.811173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2531.831561] Lustre: Skipped 1 previous similar message [ 2536.564437] LustreError: 68738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.580300] LustreError: 68738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2538.943977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2539.129732] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2539.285074] Lustre: *** cfs_fail_loc=135, val=0*** [ 2539.425118] LustreError: 71830:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2539.430801] LustreError: 71830:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.125@tcp: cfg command failed: rc = -2 [ 2539.443884] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2539.443884] [ 2539.452830] LustreError: MGC192.168.201.125@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2539.476862] LustreError: 71819:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2539.484968] LustreError: 71819:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 2539.502720] LustreError: 71819:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2539.505764] LustreError: 71819:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2539.543903] Lustre: server umount lustre-MDT0000 complete [ 2539.547452] LustreError: 71819:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2546.795397] LustreError: 68738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.822294] LustreError: 68738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2547.622041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2547.863506] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2548.259376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.330386] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2551.919731] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2553.332453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2553.397327] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2553.468780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2553.491588] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2562.530544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2562.544665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.562427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2563.552840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2563.574369] Lustre: Skipped 1 previous similar message [ 2567.294417] Lustre: server umount lustre-OST0000 complete [ 2573.792288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.793751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2573.799940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.819870] Lustre: Skipped 2 previous similar messages [ 2577.447833] Lustre: server umount lustre-MDT0000 complete [ 2578.913405] LustreError: 68743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.938376] LustreError: 68743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2581.188523] LustreError: 68723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835226 with bad export cookie 12232159101896308108 [ 2581.204418] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2581.214440] LustreError: 68723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2581.683769] Lustre: server umount lustre-MDT0001 complete [ 2594.280440] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2597.508706] Key type lgssc unregistered [ 2597.807614] LNet: 73607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2597.817270] LNetError: 73607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2597.846678] LNet: Removed LNI 192.168.201.125@tcp [ 2598.804853] Key type .llcrypt unregistered [ 2598.807175] Key type ._llcrypt unregistered [ 2625.788279] Key type ._llcrypt registered [ 2625.790029] Key type .llcrypt registered [ 2626.011754] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2627.741396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2627.870492] alg: No test for adler32 (adler32-zlib) [ 2629.125835] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2629.546646] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2631.279164] Key type lgssc registered [ 2632.359781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2643.260166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2643.318882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.415155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.522706] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2655.713212] LustreError: 75016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.723748] LustreError: 75016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2660.705424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2660.833349] LustreError: 75015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.237796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2666.863996] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2674.726930] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2681.881365] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2689.737119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2689.957366] LustreError: 76837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.184262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.666151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2697.094631] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2705.690808] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2712.459072] Lustre: Failing over lustre-MDT0000 [ 2712.546230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2712.553639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.577735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2712.933614] Lustre: server umount lustre-MDT0000 complete [ 2713.743976] LustreError: 76343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.129663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2716.145196] Lustre: Skipped 1 previous similar message [ 2718.820989] LustreError: 75011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2718.845871] LustreError: 75011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2722.046299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.184342] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2722.335487] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2722.436886] LustreError: 78104:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2722.443340] LustreError: 78104:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.125@tcp: cfg command failed: rc = -9 [ 2722.453539] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2722.453539] [ 2722.465661] LustreError: MGC192.168.201.125@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2722.483757] LustreError: 78092:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2722.491934] LustreError: 78092:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -9 [ 2722.502106] LustreError: 78092:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2722.509524] LustreError: 78092:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2722.538920] Lustre: server umount lustre-MDT0000 complete [ 2722.541193] LustreError: 78092:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2729.075298] LustreError: 75012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2729.097223] LustreError: 75012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2730.135404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.307681] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2730.317186] LustreError: 78452:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.201.125@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000c449ec5b) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2730.346767] LustreError: 75008:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2730.700031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2730.761164] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2734.191791] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2735.919855] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2736.141393] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2736.185887] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2736.269114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2746.344211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.345141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2746.363499] Lustre: Skipped 1 previous similar message [ 2751.470880] LustreError: 76837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2751.509617] LustreError: 76837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2751.610338] Lustre: server umount lustre-OST0000 complete [ 2756.577655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.583501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2756.605728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2756.624052] Lustre: Skipped 2 previous similar messages [ 2761.697491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2761.704703] Lustre: Skipped 1 previous similar message [ 2762.587979] Lustre: server umount lustre-MDT0000 complete [ 2767.063760] LustreError: 74997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835412 with bad export cookie 9551387134810104616 [ 2767.071832] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2767.081048] LustreError: 74997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2767.587605] Lustre: server umount lustre-MDT0001 complete [ 2780.199833] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2783.232629] Key type lgssc unregistered [ 2783.480855] LNet: 79887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2783.484077] LNetError: 79887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2784.549528] LNet: Removed LNI 192.168.201.125@tcp [ 2785.477863] Key type .llcrypt unregistered [ 2785.480333] Key type ._llcrypt unregistered [ 2801.961518] Key type ._llcrypt registered [ 2801.963980] Key type .llcrypt registered [ 2802.107651] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2802.396361] Key type .llcrypt unregistered [ 2802.398166] Key type ._llcrypt unregistered [ 2828.340095] Key type ._llcrypt registered [ 2828.341865] Key type .llcrypt registered [ 2828.555280] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2830.010428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2830.052935] alg: No test for adler32 (adler32-zlib) [ 2831.106464] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2831.503161] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2833.191266] Key type lgssc registered [ 2834.745306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2844.437840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2844.468981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.178910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.971711] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2856.419611] LustreError: 81783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2856.446402] LustreError: 81783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2861.551961] LustreError: 81784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2866.177534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.674330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2872.158514] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2883.711539] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2891.488844] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2899.697646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2900.163102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2907.554167] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2916.609794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2917.144858] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2927.075500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2927.083977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2927.096420] Lustre: Skipped 1 previous similar message [ 2927.096762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2927.114356] Lustre: Skipped 1 previous similar message [ 2929.016981] Lustre: server umount lustre-OST0000 complete [ 2933.216407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2933.226837] LustreError: Skipped 1 previous similar message [ 2933.234172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2933.257483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2937.313381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2937.323476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2939.226851] Lustre: server umount lustre-MDT0000 complete [ 2942.436051] LustreError: 81779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.447399] LustreError: 81779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2943.080469] LustreError: 81763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835588 with bad export cookie 13479792431217574548 [ 2943.089517] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2943.102054] LustreError: 81763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2943.488837] Lustre: server umount lustre-MDT0001 complete [ 2957.388599] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2960.554739] Key type lgssc unregistered [ 2960.800956] LNet: 85319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2960.809460] LNetError: 85319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2960.826112] LNet: Removed LNI 192.168.201.125@tcp [ 2961.892956] Key type .llcrypt unregistered [ 2961.894516] Key type ._llcrypt unregistered [ 2992.564398] Key type ._llcrypt registered [ 2992.565929] Key type .llcrypt registered [ 2992.637164] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2993.427345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2993.490602] alg: No test for adler32 (adler32-zlib) [ 2994.629843] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2995.080523] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2996.799175] Key type lgssc registered [ 2998.768144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3011.120848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3011.139237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3012.786216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3019.211986] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3022.308658] LustreError: 86732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3022.326766] LustreError: 86732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3027.442965] LustreError: 86731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3031.150182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.837447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3036.426515] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3045.591542] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3052.977479] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3061.031977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3061.472912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3068.679555] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3076.884170] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3081.991228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 3092.452294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3092.457725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.466517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3092.487475] Lustre: Skipped 1 previous similar message [ 3096.589463] Lustre: server umount lustre-OST0000 complete [ 3102.689316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3102.698296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3102.701198] Lustre: Skipped 1 previous similar message [ 3107.234625] Lustre: server umount lustre-MDT0000 complete [ 3107.814385] LustreError: 86728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.843668] LustreError: 86728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3111.119214] LustreError: 86714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835756 with bad export cookie 3595275255452763330 [ 3111.122078] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3111.132921] LustreError: 86714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3111.425629] Lustre: server umount lustre-MDT0001 complete [ 3124.101734] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 3127.445348] Key type lgssc unregistered [ 3127.807703] LNet: 90275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3127.834782] LNetError: 90275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3127.861798] LNet: Removed LNI 192.168.201.125@tcp [ 3128.749202] Key type .llcrypt unregistered [ 3128.764275] Key type ._llcrypt unregistered [ 3157.986363] Key type ._llcrypt registered [ 3157.988819] Key type .llcrypt registered [ 3158.210969] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3160.393359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3160.434693] alg: No test for adler32 (adler32-zlib) [ 3161.588947] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 3161.902499] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 3163.553274] Key type lgssc registered [ 3164.943908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3173.831410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3173.857417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.314744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3180.165708] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3185.634068] LustreError: 91688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3185.653305] LustreError: 91688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3189.565190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3189.914142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3195.284673] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3204.472551] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3211.776587] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3220.805685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3221.436462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3228.687961] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3238.668546] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3244.996377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3260.400879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3260.411346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3260.433634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3260.438501] LustreError: Skipped 1 previous similar message [ 3260.456368] Lustre: Skipped 1 previous similar message [ 3264.555778] Lustre: server umount lustre-OST0000 complete [ 3270.625447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.642400] Lustre: Skipped 2 previous similar messages [ 3270.644147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3270.660464] Lustre: Skipped 1 previous similar message [ 3274.984973] Lustre: server umount lustre-MDT0000 complete [ 3275.746174] LustreError: 94181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3275.763412] LustreError: 94181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3279.322670] LustreError: 92423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835924 with bad export cookie 1232865534666114674 [ 3279.329464] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3279.334166] LustreError: 92423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3279.584860] Lustre: server umount lustre-MDT0001 complete [ 3292.189409] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 3295.084381] Key type lgssc unregistered [ 3295.459943] LNet: 95246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3295.467206] LNetError: 95246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3295.476983] LNet: Removed LNI 192.168.201.125@tcp [ 3296.276758] Key type .llcrypt unregistered [ 3296.279840] Key type ._llcrypt unregistered [ 3322.761650] Key type ._llcrypt registered [ 3322.765813] Key type .llcrypt registered [ 3322.900459] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3324.932622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3325.013340] alg: No test for adler32 (adler32-zlib) [ 3326.440583] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 3326.857354] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 3328.743642] Key type lgssc registered [ 3330.598417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3339.702962] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3339.723894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3355.359570] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3360.479322] LustreError: 96630:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 3387.231245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3394.387465] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3403.125640] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3409.545331] Lustre: server umount lustre-OST0000 complete [ 3419.418479] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 22:54:23 (1788836063) [ 3426.568255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.895165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3431.690845] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3431.914713] LustreError: 98270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3431.935788] LustreError: 98270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3437.026407] LustreError: 98269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3441.234495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3441.538324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3446.340083] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3455.047525] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3461.848722] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3468.973148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3469.315432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.387187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3475.035392] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3484.398578] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3503.081179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3503.107388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.123536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.132938] Lustre: Skipped 1 previous similar message [ 3505.629468] Lustre: server umount lustre-OST0000 complete [ 3513.326588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.338665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.339650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3513.339657] LustreError: Skipped 1 previous similar message [ 3513.346180] Lustre: Skipped 2 previous similar messages [ 3513.396975] Lustre: Skipped 1 previous similar message [ 3515.696161] Lustre: server umount lustre-MDT0000 complete [ 3518.437580] LustreError: 98265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.459356] LustreError: 98265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3519.441248] LustreError: 98249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836164 with bad export cookie 9287601013211079572 [ 3519.443246] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3519.450179] LustreError: 98249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3519.665458] Lustre: server umount lustre-MDT0001 complete [ 3530.723024] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 3533.284651] Key type lgssc unregistered [ 3533.650580] LNet: 101959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3533.658596] LNetError: 101959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3533.677831] LNet: Removed LNI 192.168.201.125@tcp [ 3534.522284] Key type .llcrypt unregistered [ 3534.524795] Key type ._llcrypt unregistered [ 3581.969125] Key type ._llcrypt registered [ 3581.970826] Key type .llcrypt registered [ 3582.084891] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3583.624793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3583.760078] alg: No test for adler32 (adler32-zlib) [ 3585.015199] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 3585.319652] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 3586.999250] Key type lgssc registered [ 3588.285258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3598.936280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3598.951736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3600.420558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.700061] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3610.598895] LustreError: 104145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3610.611334] LustreError: 104145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3615.377184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3615.739483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3620.386307] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3629.498678] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3636.111374] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3645.090137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3645.413255] LustreError: 105965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.704467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3651.056292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3654.933670] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3666.576901] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3676.647303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3676.662824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3676.684349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3680.737641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3680.743452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3682.413543] Lustre: server umount lustre-OST0000 complete [ 3687.396943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3687.405475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3687.416383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3687.418612] Lustre: Skipped 1 previous similar message [ 3690.977351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3690.979907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3691.003605] Lustre: Skipped 1 previous similar message [ 3693.036321] Lustre: server umount lustre-MDT0000 complete [ 3696.097062] LustreError: 104144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3696.125058] LustreError: 104144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3697.440661] LustreError: 105988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836342 with bad export cookie 10918674877332302065 [ 3697.443888] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3697.471434] LustreError: 105988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3697.842971] Lustre: server umount lustre-MDT0001 complete [ 3709.486949] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 3712.325228] Key type lgssc unregistered [ 3712.620787] LNet: 107680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3712.628257] LNetError: 107680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3712.647562] LNet: Removed LNI 192.168.201.125@tcp [ 3713.599822] Key type .llcrypt unregistered [ 3713.602883] Key type ._llcrypt unregistered [ 3735.908029] Key type ._llcrypt registered [ 3735.909494] Key type .llcrypt registered [ 3736.068489] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3737.152052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3737.458855] alg: No test for adler32 (adler32-zlib) [ 3738.675609] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 3738.973096] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 3740.687665] Key type lgssc registered [ 3741.843306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3750.109532] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3750.119165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3765.600439] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3770.719584] LustreError: 108920:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 3797.471216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3803.904937] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3812.079663] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3819.195891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.485085] LustreError: MGC192.168.201.125@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3819.498819] LustreError: 109954:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3819.507138] LustreError: 109954:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 3819.522095] LustreError: 109954:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3819.615418] Lustre: server umount lustre-MDT0000 complete [ 3819.618575] LustreError: 109954:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3836.569554] Lustre: server umount lustre-OST0000 complete [ 3847.636186] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 3859.397321] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3902.980927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3903.189254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3903.254588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3903.350529] Lustre: lustre-MDT0000: new disk, initializing [ 3903.434401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3903.456392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3907.758210] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3918.462149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.699685] Lustre: 112748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3918.826707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3918.834861] Lustre: Skipped 1 previous similar message [ 3918.941896] Lustre: lustre-MDT0001: new disk, initializing [ 3919.135277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3919.179966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3919.203635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3924.479228] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3935.494757] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3942.453985] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3951.795698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3952.097909] Lustre: lustre-OST0000: new disk, initializing [ 3952.104135] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3952.111225] Lustre: 114717:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3952.219601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3959.480903] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3960.441050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3960.463283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3960.692605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3971.504201] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3977.064344] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3977.382536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3983.478608] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3983.957279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3991.521307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3991.526879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3991.548127] Lustre: Skipped 1 previous similar message [ 3991.566143] Lustre: Skipped 1 previous similar message [ 3993.066506] Lustre: server umount lustre-OST0000 complete [ 4001.249026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4001.266520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.307123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4004.546798] Lustre: server umount lustre-MDT0000 complete [ 4009.553613] LustreError: 112740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836654 with bad export cookie 9820729490597040952 [ 4009.561131] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4009.564409] LustreError: 112740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4009.991935] Lustre: server umount lustre-MDT0001 complete [ 4019.220839] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 23:04:23 (1788836663) [ 4023.043841] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4042.388759] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 4052.887194] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4094.478803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.755313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4094.788945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4094.883970] Lustre: lustre-MDT0000: new disk, initializing [ 4094.935563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.973539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4101.149561] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4111.298608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4111.409461] Lustre: 119624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4111.451444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4111.456207] Lustre: Skipped 1 previous similar message [ 4111.560150] Lustre: lustre-MDT0001: new disk, initializing [ 4111.665534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4111.708900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4111.731068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4116.253774] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4127.444680] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4134.148556] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4142.011705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4142.410347] Lustre: lustre-OST0000: new disk, initializing [ 4142.416750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4142.434785] Lustre: 121591:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4142.552716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4150.185073] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4150.924972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4150.935175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4151.157124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4161.872865] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4166.773915] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4167.072460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4171.463262] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4171.673153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4176.865160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4176.873483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4176.891664] Lustre: Skipped 1 previous similar message [ 4176.899931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4176.919270] Lustre: Skipped 1 previous similar message [ 4179.784693] Lustre: server umount lustre-OST0000 complete [ 4187.105033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.118692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4187.137374] Lustre: Skipped 2 previous similar messages [ 4187.142322] Lustre: Skipped 1 previous similar message [ 4191.414496] Lustre: server umount lustre-MDT0000 complete [ 4192.234080] LustreError: 119630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.257158] LustreError: 119630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4195.425806] LustreError: 119615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836840 with bad export cookie 9820729490597041841 [ 4195.435967] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4195.438053] LustreError: 119615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4195.865578] Lustre: server umount lustre-MDT0001 complete [ 4206.104643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4206.650611] LustreError: 123810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4206.756868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.788660] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4212.193584] LustreError: 123811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4216.292748] LustreError: 123810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.343967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.411219] LustreError: 123811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4226.086043] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4235.987178] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4242.591149] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4249.676152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4255.951678] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4263.334180] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4273.401718] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4281.827196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4281.829324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.838237] LustreError: Skipped 1 previous similar message [ 4281.844420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4281.852922] Lustre: Skipped 1 previous similar message [ 4281.880512] Lustre: Skipped 2 previous similar messages [ 4284.172282] Lustre: server umount lustre-OST0000 complete [ 4288.480625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4292.064783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4292.070226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4292.079062] Lustre: Skipped 1 previous similar message [ 4292.103410] Lustre: Skipped 1 previous similar message [ 4294.871552] Lustre: server umount lustre-MDT0000 complete [ 4297.195838] LustreError: 124831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4297.216647] LustreError: 124831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4298.700861] LustreError: 123790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836943 with bad export cookie 9820729490597042723 [ 4298.705401] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4298.711551] LustreError: 123790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4299.029392] Lustre: server umount lustre-MDT0001 complete [ 4309.727185] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4312.243610] Key type lgssc unregistered [ 4312.612903] LNet: 127439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4312.623763] LNetError: 127439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4312.638386] LNet: Removed LNI 192.168.201.125@tcp [ 4313.820983] Key type .llcrypt unregistered [ 4313.823477] Key type ._llcrypt unregistered [ 4335.496190] Key type ._llcrypt registered [ 4335.513261] Key type .llcrypt registered [ 4335.619772] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 4353.187992] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4354.785892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4354.817852] alg: No test for adler32 (adler32-zlib) [ 4356.042281] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 4356.381817] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4358.072048] Key type lgssc registered [ 4359.340210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4400.279135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4400.349344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4401.692749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4401.728478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4401.828108] Lustre: lustre-MDT0000: new disk, initializing [ 4401.909215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4401.938841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4406.860590] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4419.165806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4419.281318] Lustre: 130591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4419.398160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4419.411426] Lustre: Skipped 1 previous similar message [ 4419.505623] Lustre: lustre-MDT0001: new disk, initializing [ 4419.612343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4419.667923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4419.681362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4424.795346] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4435.560560] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4442.339763] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4451.486497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4451.934729] Lustre: lustre-OST0000: new disk, initializing [ 4451.937905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4451.942050] Lustre: 132561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4452.043871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4460.035650] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4460.642981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4460.652347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4460.749337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4471.873618] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4477.529501] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4477.778951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4482.744364] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4482.980245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4486.627807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4486.639213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4486.640742] LustreError: Skipped 1 previous similar message [ 4486.666725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4491.547245] Lustre: server umount lustre-OST0000 complete [ 4496.354545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4496.361657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4496.380590] Lustre: Skipped 1 previous similar message [ 4496.391608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4496.394214] Lustre: Skipped 1 previous similar message [ 4501.988122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4502.002528] Lustre: Skipped 2 previous similar messages [ 4502.743419] Lustre: server umount lustre-MDT0000 complete [ 4506.084272] LustreError: 130603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4506.125328] LustreError: 130603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4507.427755] LustreError: 130584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837152 with bad export cookie 14801640926149744979 [ 4507.431806] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4507.436729] LustreError: 130584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4507.807558] Lustre: server umount lustre-MDT0001 complete [ 4519.200143] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 23:12:43 (1788837163) [ 4527.495815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4528.255434] LustreError: 134978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.327107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4533.289870] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4533.729677] LustreError: 134977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4538.851292] LustreError: 134978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4542.937586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4543.405562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4548.156925] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4555.785929] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4563.156350] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4568.865804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4568.876367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4568.890015] Lustre: Skipped 2 previous similar messages [ 4568.923983] Lustre: Skipped 1 previous similar message [ 4574.177551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4574.187098] Lustre: Skipped 1 previous similar message [ 4580.320158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4580.531829] Lustre: server umount lustre-MDT0000 complete [ 4584.419459] LustreError: 134977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4584.438933] LustreError: 134977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4584.790920] LustreError: 135711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837229 with bad export cookie 14801640926149745861 [ 4584.799951] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4584.809576] LustreError: 135711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4585.202455] Lustre: server umount lustre-MDT0001 complete [ 4596.699959] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 23:14:00 (1788837240) [ 4604.991676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4620.575750] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4625.695754] LustreError: 137396:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 4651.423291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4659.659144] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4669.347750] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4671.664233] Lustre: server umount lustre-OST0000 complete [ 4682.869877] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 23:15:27 (1788837327) [ 4690.563604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4691.266261] LustreError: 138844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4691.463414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4696.593983] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4707.376377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4712.291270] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4720.546263] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4727.104669] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4734.984713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4735.438586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4735.444679] Lustre: Skipped 1 previous similar message [ 4743.200928] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4754.662569] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4763.109488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4763.113532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4763.145590] Lustre: Skipped 2 previous similar messages [ 4763.156900] Lustre: Skipped 4 previous similar messages [ 4769.446426] Lustre: server umount lustre-MDT0000 complete [ 4773.359718] LustreError: 138845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4773.376931] LustreError: 138845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4773.749285] LustreError: 138826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837418 with bad export cookie 14801640926149746526 [ 4773.762874] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4774.089083] Lustre: server umount lustre-MDT0001 complete [ 4788.864673] Lustre: server umount lustre-OST0000 complete [ 4800.341046] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 23:17:24 (1788837444) [ 4809.293729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4810.163252] LustreError: 142502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4810.324334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4815.882808] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4825.918129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4831.975839] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4840.916358] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4847.438852] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4855.169164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4862.864387] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4872.030297] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4877.022839] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4880.181904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4880.491539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4885.617519] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4885.900432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4890.592464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4890.602093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4890.609205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4890.609212] Lustre: Skipped 3 previous similar messages [ 4890.635219] Lustre: Skipped 1 previous similar message [ 4902.367300] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4902.539869] Lustre: server umount lustre-OST0000 complete [ 4908.513746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4908.518496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4912.627808] Lustre: server umount lustre-MDT0000 complete [ 4916.206510] LustreError: 143231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4916.245264] LustreError: 143231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4917.501987] LustreError: 142483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837562 with bad export cookie 14801640926149747597 [ 4917.517021] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4917.523821] LustreError: 142483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4918.159247] Lustre: server umount lustre-MDT0001 complete [ 4928.872795] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 23:19:32 (1788837572) [ 4937.492346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4953.318426] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4958.434150] LustreError: 146684:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 4984.159421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4984.169031] Lustre: Skipped 2 previous similar messages [ 4990.116450] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4997.253379] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5004.993622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.747951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5011.403636] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5020.288729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5024.799108] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5032.154075] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5038.359798] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5043.686454] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5043.972942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5048.280581] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5048.505469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5051.370076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5051.372273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5051.375810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5051.375819] Lustre: Skipped 8 previous similar messages [ 5051.398816] Lustre: Skipped 3 previous similar messages [ 5056.624630] Lustre: server umount lustre-MDT0000 complete [ 5060.258227] LustreError: 146692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837705 with bad export cookie 14801640926149748619 [ 5060.260655] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5060.267287] LustreError: 146692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5060.572464] Lustre: server umount lustre-MDT0001 complete [ 5076.056121] Lustre: server umount lustre-OST0000 complete [ 5086.297513] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 23:22:10 (1788837730) [ 5093.789669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5109.279902] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5114.400281] LustreError: 150881:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 5146.379416] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5153.927411] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5160.399436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5160.836096] LustreError: 150906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5160.852273] LustreError: 150906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5160.979296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5165.570621] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5174.816909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5181.687499] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5190.890883] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5197.582945] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5204.699564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5204.913476] Lustre: lustre-OST0001: new disk, initializing [ 5204.923937] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5204.933167] Lustre: 153752:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5211.745737] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5214.808149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5214.821278] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5214.965597] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5222.955437] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5228.515376] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5228.695259] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5234.182537] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5234.375499] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5236.710292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5236.719713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5236.734459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5236.736990] Lustre: Skipped 5 previous similar messages [ 5242.810841] Lustre: server umount lustre-OST0000 complete [ 5253.502041] Lustre: server umount lustre-OST0001 complete [ 5263.954253] Lustre: server umount lustre-MDT0000 complete [ 5267.780558] LustreError: 150887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837912 with bad export cookie 14801640926149749571 [ 5267.784638] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5267.790335] LustreError: 150887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5268.469763] Lustre: server umount lustre-MDT0001 complete [ 5293.792681] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 23:25:38 (1788837938) [ 5295.555318] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5297.301628] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5298.951716] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 23:25:43 (1788837943) [ 5307.900794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5308.362786] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5308.407803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5308.646954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5308.655928] Lustre: Skipped 6 previous similar messages [ 5313.943459] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5323.146536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.269180] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5323.322313] Lustre: 157314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5328.836612] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5337.287287] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5343.446675] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5350.519478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5350.682315] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5357.951927] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5359.177880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5366.317820] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5371.686186] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5371.939982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5377.015833] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5377.237550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5379.552562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5379.556025] LustreError: Skipped 3 previous similar messages [ 5379.557133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5379.567763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5379.567772] Lustre: Skipped 7 previous similar messages [ 5379.612449] Lustre: Skipped 6 previous similar messages [ 5385.522539] Lustre: server umount lustre-OST0000 complete [ 5406.587462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5408.942308] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.961207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.969718] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5413.542236] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5421.811474] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5428.465682] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5428.620481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5433.346443] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5433.614402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5441.577893] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa00808033000.ost_server_uuid 50 [ 5443.502955] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa00808033000.ost_server_uuid in FULL state after 0 sec [ 5443.710710] LustreError: 158583:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.25@tcp) returned error from blocking AST (req@000000008d3424cc x1875731547436416 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000021050442/0xcd69fdfff80a349d lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.25@tcp remote: 0x34b5124e4492e52 expref: 5 pid: 157324 timeout: 5543 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5443.746558] LustreError: lustre-MDT0000: A client on nid 192.168.201.25@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5443.753896] LustreError: 157312:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.25@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000021050442/0xcd69fdfff80a349d lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.25@tcp remote: 0x34b5124e4492e52 expref: 6 pid: 157324 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5450.221522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5450.243147] LustreError: Skipped 1 previous similar message [ 5455.621353] Lustre: server umount lustre-OST0000 complete [ 5466.862609] Lustre: server umount lustre-MDT0000 complete [ 5470.697801] LustreError: 157327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5470.731856] LustreError: 157327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5471.602116] LustreError: 157309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838116 with bad export cookie 14801640926149750754 [ 5471.609075] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5471.620866] LustreError: 157309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5478.114392] Lustre: server umount lustre-MDT0001 complete [ 5489.210342] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 5492.291849] Key type lgssc unregistered [ 5492.596903] LNet: 163429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5492.610625] LNetError: 163429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5493.678124] LNet: Removed LNI 192.168.201.125@tcp [ 5494.623515] Key type .llcrypt unregistered [ 5494.625921] Key type ._llcrypt unregistered [ 5521.334990] Key type ._llcrypt registered [ 5521.337758] Key type .llcrypt registered [ 5521.508021] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5523.204706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5523.301162] alg: No test for adler32 (adler32-zlib) [ 5524.488706] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 5524.744996] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 5526.463657] Key type lgssc registered [ 5528.299493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5537.682967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5537.696995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5539.034558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.628626] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5549.538711] LustreError: 164839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5549.562113] LustreError: 164839:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5554.194327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5554.679549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5558.954421] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5567.708239] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5574.916911] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5581.892139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5582.249104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.309598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5589.754070] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5597.543814] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5604.793057] Lustre: Failing over lustre-MDT0000 [ 5604.846837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5604.878457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5605.311924] Lustre: server umount lustre-MDT0000 complete [ 5606.025116] LustreError: 164835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5609.952710] LustreError: 166057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5609.977546] LustreError: 166057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5615.074455] LustreError: 164840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5615.120963] LustreError: 164840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5617.802700] LustreError: 164820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.25@tcp arrived at 1788838262 with bad export cookie 2052038201901434216 [ 5617.820106] LustreError: 164820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5620.204160] LustreError: 164839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5620.229069] LustreError: 164839:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5626.341178] Lustre: 164150:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838255/real 1788838255] req@000000004da59bce x1875732771924480/t0(0) o400->MGC192.168.201.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788838271 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5626.389748] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5627.408723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5631.458271] LustreError: 166057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5631.472828] LustreError: 166057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5636.963286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5637.019950] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5642.212450] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5642.216627] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5642.247203] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5644.879620] Lustre: lustre-MDT0000: Denying connection for new client a66a09c0-a394-4010-aa5e-7b5934808d8a (at 192.168.201.25@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5650.040539] Lustre: lustre-MDT0000: Denying connection for new client a66a09c0-a394-4010-aa5e-7b5934808d8a (at 192.168.201.25@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 5656.033492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5656.046994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.065455] Lustre: Skipped 2 previous similar messages [ 5656.085400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5656.091241] Lustre: Skipped 2 previous similar messages [ 5657.570774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5657.572868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5657.621986] Lustre: Skipped 1 previous similar message [ 5659.902731] Lustre: server umount lustre-OST0000 complete [ 5665.019264] LustreError: 168571:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5665.027815] Lustre: 167920:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5665.037785] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5665.063775] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5665.075472] LustreError: 167920:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f7736b17 x1875732771945216/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5665.088537] LustreError: 167920:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5665.095753] LustreError: 167920:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5665.106360] Lustre: lustre-MDT0000: Recovery over after 0:23, of 2 clients 0 recovered and 2 were evicted. [ 5665.257193] Lustre: server umount lustre-MDT0000 complete [ 5667.808257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5667.826407] LustreError: 164835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5667.853381] LustreError: 164835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5669.451773] LustreError: 164820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838314 with bad export cookie 2052038201901434321 [ 5669.457056] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5669.470240] LustreError: 164820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5672.012226] Lustre: server umount lustre-MDT0001 complete [ 5684.443571] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 5688.035409] Key type lgssc unregistered [ 5688.308317] LNet: 169322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5688.322034] LNetError: 169322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5688.348237] LNet: Removed LNI 192.168.201.125@tcp [ 5689.451256] Key type .llcrypt unregistered [ 5689.453228] Key type ._llcrypt unregistered [ 5714.595884] Key type ._llcrypt registered [ 5714.599431] Key type .llcrypt registered [ 5714.769727] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5715.536334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5715.616683] alg: No test for adler32 (adler32-zlib) [ 5716.694628] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 5716.990224] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 5718.671169] Key type lgssc registered [ 5719.739602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5729.166613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5729.190758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5730.734449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5735.850568] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5740.005284] LustreError: 170732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5740.048959] LustreError: 170732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5745.127863] LustreError: 170733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.801437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5746.278159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5751.463704] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5760.227710] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5767.813031] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5777.049419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5777.756504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5785.691807] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5795.710512] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5800.373376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5810.660873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5810.684702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5810.693869] Lustre: Skipped 1 previous similar message [ 5810.710474] Lustre: Skipped 1 previous similar message [ 5813.486958] Lustre: server umount lustre-OST0000 complete [ 5817.824507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5817.828112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5817.835914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5820.899715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5820.900510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.665255] Lustre: server umount lustre-MDT0000 complete [ 5826.029502] LustreError: 171463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5827.735882] LustreError: 170713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838472 with bad export cookie 5234302280712472839 [ 5827.751959] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5828.241556] Lustre: server umount lustre-MDT0001 complete [ 5839.806304] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 5842.496750] Key type lgssc unregistered [ 5842.759760] LNet: 174269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5842.765978] LNetError: 174269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5842.781176] LNet: Removed LNI 192.168.201.125@tcp [ 5843.566588] Key type .llcrypt unregistered [ 5843.568597] Key type ._llcrypt unregistered [ 5880.686935] Key type ._llcrypt registered [ 5880.695722] Key type .llcrypt registered [ 5880.874672] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5882.895351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5882.960234] alg: No test for adler32 (adler32-zlib) [ 5884.050979] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 5884.318053] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 5885.978301] Key type lgssc registered [ 5887.467952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5896.704217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5896.742615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5898.193505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5902.907733] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5908.450140] LustreError: 175970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5908.476151] LustreError: 175970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5913.469105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5913.576856] LustreError: 175969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5914.142805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5919.126202] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5927.709557] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5934.169051] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5942.318124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5942.871739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5943.947673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5951.325181] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5960.740843] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5969.889171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5969.898362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5969.915390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5969.954303] Lustre: Skipped 1 previous similar message [ 5975.083382] Lustre: server umount lustre-OST0000 complete [ 5980.128649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5980.132716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5980.166983] Lustre: Skipped 1 previous similar message [ 5980.196605] Lustre: Skipped 2 previous similar messages [ 5985.249032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5985.458359] Lustre: server umount lustre-MDT0000 complete [ 5989.366649] LustreError: 175950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838634 with bad export cookie 13442736975579949311 [ 5989.372866] LustreError: 175950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5989.373668] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5989.747657] Lustre: server umount lustre-MDT0001 complete [ 6001.908193] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 6004.655877] Key type lgssc unregistered [ 6004.913111] LNet: 179504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6004.927357] LNetError: 179504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6004.946920] LNet: Removed LNI 192.168.201.125@tcp [ 6006.109312] Key type .llcrypt unregistered [ 6006.112453] Key type ._llcrypt unregistered [ 6028.162569] Key type ._llcrypt registered [ 6028.164178] Key type .llcrypt registered [ 6028.301972] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6029.282551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6029.310413] alg: No test for adler32 (adler32-zlib) [ 6030.392143] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 6030.688342] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 6032.424047] Key type lgssc registered [ 6033.672978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6044.941883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6044.976688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6046.367245] Lustre: *** cfs_fail_loc=135, val=0*** [ 6046.474849] LustreError: 181003:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6046.478775] LustreError: 181003:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.125@tcp: cfg command failed: rc = -2 [ 6046.484621] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6046.484621] [ 6046.491855] LustreError: MGC192.168.201.125@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6046.500925] LustreError: 180965:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6046.505711] LustreError: 180965:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 6046.510598] LustreError: 180965:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 6046.515436] LustreError: 180965:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6046.576219] Lustre: server umount lustre-MDT0000 complete [ 6046.578404] LustreError: 180965:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6052.706976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6053.078259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6057.087862] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6063.598732] LustreError: 181376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6063.617728] LustreError: 181376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6066.129083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6066.520811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6071.606938] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6079.102622] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6086.570951] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6091.363269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.369551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6091.373776] Lustre: Skipped 1 previous similar message [ 6091.393926] Lustre: Skipped 1 previous similar message [ 6095.836566] Lustre: server umount lustre-MDT0000 complete [ 6096.865077] LustreError: 182617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6096.895760] LustreError: 182617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6101.212348] LustreError: 181355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838746 with bad export cookie 17244177484619709060 [ 6101.226034] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6101.228912] LustreError: 181355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6101.993506] Lustre: server umount lustre-MDT0001 complete [ 6115.715326] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 6119.731430] Key type lgssc unregistered [ 6120.133426] LNet: 183699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6120.140304] LNetError: 183699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6120.165996] LNet: Removed LNI 192.168.201.125@tcp [ 6121.538796] Key type .llcrypt unregistered [ 6121.542143] Key type ._llcrypt unregistered [ 6144.631650] Key type ._llcrypt registered [ 6144.635078] Key type .llcrypt registered [ 6144.732764] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 6145.031507] Key type .llcrypt unregistered [ 6145.033183] Key type ._llcrypt unregistered [ 6162.961690] Key type ._llcrypt registered [ 6162.966698] Key type .llcrypt registered [ 6163.120952] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6164.564059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6164.683609] alg: No test for adler32 (adler32-zlib) [ 6165.945831] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 6166.233861] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 6167.889163] Key type lgssc registered [ 6169.337915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6177.839182] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6177.858760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6193.379445] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6198.495394] LustreError: 185823:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 6225.313293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6231.958496] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6240.206578] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6247.667919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6248.110916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6248.234636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6253.500252] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6253.544035] LustreError: 185848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6253.570324] LustreError: 185848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6258.659406] LustreError: 185849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6260.712954] LustreError: 186862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6260.733996] LustreError: 186862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6263.049328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6263.539532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6268.573518] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6277.310747] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6284.606734] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6288.022361] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6299.106384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6299.123727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6299.149177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6299.617394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6304.747783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6304.760256] Lustre: Skipped 1 previous similar message [ 6309.877364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6309.887252] Lustre: Skipped 1 previous similar message [ 6311.391790] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6311.651996] Lustre: server umount lustre-OST0000 complete [ 6320.099105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6320.101054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6320.112763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6320.112774] Lustre: Skipped 1 previous similar message [ 6320.141596] Lustre: Skipped 2 previous similar messages [ 6321.836777] Lustre: server umount lustre-MDT0000 complete [ 6325.215967] LustreError: 186866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.225455] LustreError: 186866:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6325.381136] LustreError: 185831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838970 with bad export cookie 7633722937124278600 [ 6325.390569] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6325.697786] Lustre: server umount lustre-MDT0001 complete [ 6335.530309] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 6337.583495] Key type lgssc unregistered [ 6337.841526] LNet: 189741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6337.850343] LNetError: 189741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6337.869845] LNet: Removed LNI 192.168.201.125@tcp [ 6338.598420] Key type .llcrypt unregistered [ 6338.600506] Key type ._llcrypt unregistered [ 6361.053454] Key type ._llcrypt registered [ 6361.056561] Key type .llcrypt registered [ 6361.192416] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6362.440502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6362.469652] alg: No test for adler32 (adler32-zlib) [ 6363.434910] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 6363.637307] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 6365.303826] Key type lgssc registered [ 6366.548583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6375.390420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6375.432550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6377.120428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6382.757881] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6387.691247] LustreError: 191131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6387.714882] LustreError: 191131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6392.806165] LustreError: 191132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6392.862148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6393.312177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6397.765126] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6405.759700] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6412.134397] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6419.688357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6420.163383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6422.215980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6426.415794] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6434.906964] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6441.044855] Lustre: Failing over lustre-MDT0000 [ 6441.442892] Lustre: server umount lustre-MDT0000 complete [ 6442.654497] LustreError: 192373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6444.012339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6444.034663] Lustre: Skipped 2 previous similar messages [ 6445.609273] LustreError: 191128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6445.639177] LustreError: 191128:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6450.787981] LustreError: 192373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6450.811881] LustreError: 192373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6458.846384] LustreError: 191127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6458.883892] LustreError: 191127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6460.256218] Lustre: 190441:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788839089/real 1788839089] req@00000000756e4339 x1875733651680640/t0(0) o400->MGC192.168.201.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788839105 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6460.284890] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6461.365888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6470.935936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6471.001529] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6474.354284] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6476.281139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6476.314550] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6476.391796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6476.503128] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6481.658534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6493.166132] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6500.322484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6500.329768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6500.346325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6501.856365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6501.859081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6501.895863] Lustre: Skipped 1 previous similar message [ 6506.978280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6506.987935] Lustre: Skipped 1 previous similar message [ 6512.098393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6512.107446] Lustre: Skipped 1 previous similar message [ 6514.144945] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6514.362569] Lustre: server umount lustre-OST0000 complete [ 6522.337448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6522.340591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6522.360400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6522.370028] Lustre: Skipped 1 previous similar message [ 6522.394545] Lustre: Skipped 1 previous similar message [ 6532.587576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6532.595160] Lustre: Skipped 3 previous similar messages [ 6533.088160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6533.273514] Lustre: server umount lustre-MDT0000 complete [ 6536.661962] LustreError: 191111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839181 with bad export cookie 15944863395707260606 [ 6536.679692] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6536.990139] Lustre: server umount lustre-MDT0001 complete [ 6550.222760] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 6552.800546] Key type lgssc unregistered [ 6553.167767] LNet: 196072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6553.176684] LNetError: 196072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6554.220667] LNet: Removed LNI 192.168.201.125@tcp [ 6554.925391] Key type .llcrypt unregistered [ 6554.928494] Key type ._llcrypt unregistered [ 6577.934471] Key type ._llcrypt registered [ 6577.937149] Key type .llcrypt registered [ 6578.022491] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6579.012575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6579.147479] alg: No test for adler32 (adler32-zlib) [ 6580.315344] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 6580.572637] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 6582.295334] Key type lgssc registered [ 6583.351460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6591.559727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6591.584093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6593.154671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6598.265420] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6603.750832] LustreError: 197479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6603.785574] LustreError: 197479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6607.282219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6607.583817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6612.600789] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6620.346807] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6626.229077] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6635.664807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6636.094938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6638.153408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6644.222745] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6654.308520] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6658.444842] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6662.205828] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6682.700602] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6693.863732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6693.876199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6693.908477] Lustre: Skipped 1 previous similar message [ 6693.913461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6693.917234] Lustre: Skipped 1 previous similar message [ 6696.940949] Lustre: server umount lustre-OST0000 complete [ 6704.097278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6704.102829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6704.121347] Lustre: Skipped 1 previous similar message [ 6704.132026] Lustre: Skipped 1 previous similar message [ 6708.037420] Lustre: server umount lustre-MDT0000 complete [ 6709.217091] LustreError: 197478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.232564] LustreError: 197478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6712.203060] LustreError: 197458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839357 with bad export cookie 13068800863923913366 [ 6712.214713] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6712.223661] LustreError: 197458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6712.599557] Lustre: server umount lustre-MDT0001 complete [ 6724.085825] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 6727.661786] Key type lgssc unregistered [ 6728.000855] LNet: 201345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6728.011542] LNetError: 201345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6728.033222] LNet: Removed LNI 192.168.201.125@tcp [ 6728.859519] Key type .llcrypt unregistered [ 6728.863183] Key type ._llcrypt unregistered [ 6748.868660] Key type ._llcrypt registered [ 6748.871189] Key type .llcrypt registered [ 6749.056134] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6750.379369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6750.524132] alg: No test for adler32 (adler32-zlib) [ 6752.025397] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 6752.521714] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 6754.255245] Key type lgssc registered [ 6755.636859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6766.037812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6766.056694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.597026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6773.406979] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6777.826047] LustreError: 202611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6777.841317] LustreError: 202611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6782.671862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.951248] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6783.118026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6787.362716] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6795.935657] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6802.211896] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6810.502290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6810.811408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6812.855585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6819.279719] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6830.087923] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6837.336063] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6850.374386] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6857.173500] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6861.943724] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6874.594581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6874.609733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6874.617314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6874.631578] Lustre: Skipped 1 previous similar message [ 6879.714036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6879.728148] Lustre: Skipped 2 previous similar messages [ 6880.921079] Lustre: server umount lustre-OST0000 complete [ 6889.953343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6889.973934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6889.985399] Lustre: Skipped 1 previous similar message [ 6892.515884] Lustre: server umount lustre-MDT0000 complete [ 6895.072576] LustreError: 202606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6895.093261] LustreError: 202606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6897.228791] LustreError: 205294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839542 with bad export cookie 4391562405631835989 [ 6897.237418] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6897.244518] LustreError: 205294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6897.831733] Lustre: server umount lustre-MDT0001 complete [ 6911.647304] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 6915.328867] Key type lgssc unregistered [ 6915.801679] LNet: 207347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6915.815042] LNetError: 207347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6915.827862] LNet: Removed LNI 192.168.201.125@tcp [ 6917.019246] Key type .llcrypt unregistered [ 6917.030870] Key type ._llcrypt unregistered [ 6946.901611] Key type ._llcrypt registered [ 6946.904553] Key type .llcrypt registered [ 6947.070667] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6948.247781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6948.273144] alg: No test for adler32 (adler32-zlib) [ 6949.631392] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 6949.864681] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 6951.603221] Key type lgssc registered [ 6953.257309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6962.813515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6962.827706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6964.414366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6969.766988] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6973.923421] LustreError: 208756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6973.948524] LustreError: 208756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6979.058548] LustreError: 208757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6981.656400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6982.526945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6989.702161] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6998.932620] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7006.773333] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7015.662375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7016.176074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7025.105260] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7034.607881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7034.650571] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7054.304473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7054.320769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7054.345260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7055.338377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7055.356909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7055.357956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7055.398722] Lustre: Skipped 1 previous similar message [ 7057.956336] Lustre: server umount lustre-OST0000 complete [ 7064.550550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7064.560414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7064.587132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7070.690494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7070.717054] Lustre: Skipped 3 previous similar messages [ 7075.828568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7075.837278] Lustre: Skipped 1 previous similar message [ 7076.319127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7076.582791] Lustre: server umount lustre-MDT0000 complete [ 7079.909152] LustreError: 208757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7079.939203] LustreError: 208757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7081.531124] LustreError: 208738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839726 with bad export cookie 3969391038635836874 [ 7081.548143] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7081.561555] LustreError: 208738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7082.412845] Lustre: server umount lustre-MDT0001 complete [ 7098.951519] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 7102.716404] Key type lgssc unregistered [ 7103.252731] LNet: 212444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7103.266750] LNetError: 212444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7103.287886] LNet: Removed LNI 192.168.201.125@tcp [ 7104.214794] Key type .llcrypt unregistered [ 7104.217516] Key type ._llcrypt unregistered [ 7128.646189] Key type ._llcrypt registered [ 7128.654485] Key type .llcrypt registered [ 7128.852360] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 7130.504278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7130.570992] alg: No test for adler32 (adler32-zlib) [ 7131.704877] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 7132.084024] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 7133.847290] Key type lgssc registered [ 7135.130692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7143.997272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7144.025656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7145.740841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7150.486166] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7155.184261] LustreError: 213722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.213163] LustreError: 213722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7160.291710] LustreError: 213723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7162.762138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.614459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7169.767672] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7181.419401] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7188.786743] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7196.254519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7196.705863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7204.025760] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7213.736557] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7215.174282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7238.632239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7238.640333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7238.661514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7240.674044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7240.677552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7240.727633] Lustre: Skipped 1 previous similar message [ 7242.905723] Lustre: server umount lustre-OST0000 complete [ 7250.406781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7250.412929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7250.421499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7254.532458] Lustre: server umount lustre-MDT0000 complete [ 7256.036075] LustreError: 213719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7256.076113] LustreError: 213719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7258.586228] LustreError: 213705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839903 with bad export cookie 14598333521733916619 [ 7258.596689] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7258.604636] LustreError: 213705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7259.223524] Lustre: server umount lustre-MDT0001 complete [ 7273.730627] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 7277.205204] Key type lgssc unregistered [ 7277.668389] LNet: 217429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7277.684391] LNetError: 217429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7277.703862] LNet: Removed LNI 192.168.201.125@tcp [ 7278.777855] Key type .llcrypt unregistered [ 7278.783529] Key type ._llcrypt unregistered [ 7303.522719] Key type ._llcrypt registered [ 7303.526353] Key type .llcrypt registered [ 7303.639961] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 7304.931128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7304.982096] alg: No test for adler32 (adler32-zlib) [ 7306.043511] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 7306.312348] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 7307.951178] Key type lgssc registered [ 7308.982805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7318.067410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7318.117866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7319.584570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7325.089729] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7329.767128] LustreError: 218689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7329.784344] LustreError: 218689:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7334.894062] LustreError: 218690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7334.981779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7335.364870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7340.369631] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7349.238620] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7358.166820] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7366.189266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7366.630933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7372.606704] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7381.555869] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7389.189695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7401.876617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7401.979248] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7402.232145] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7409.755313] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7419.349820] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7421.557935] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7433.500276] Lustre: Permanently deactivating lustre-OST0001 [ 7433.525252] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7440.473924] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7447.481509] Lustre: Permanently reactivating lustre-OST0001 [ 7447.491957] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7447.495352] Lustre: Skipped 2 previous similar messages [ 7448.790445] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7448.802843] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7448.808586] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7448.818129] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7448.849111] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7458.279966] Lustre: server umount lustre-OST0001 complete [ 7459.303499] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7459.335083] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.353969] LustreError: 222363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7459.372068] Lustre: Skipped 2 previous similar messages [ 7459.430790] LustreError: 222363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7465.960514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7465.964101] LustreError: 220516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7465.964731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7465.964737] LustreError: Skipped 1 previous similar message [ 7465.965656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7466.022498] Lustre: Skipped 1 previous similar message [ 7466.120088] LustreError: 220516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7471.263856] Lustre: server umount lustre-OST0000 complete [ 7476.199190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7476.226826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7476.233740] Lustre: Skipped 2 previous similar messages [ 7481.312869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7482.876541] Lustre: server umount lustre-MDT0000 complete [ 7486.437195] LustreError: 218690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7486.468642] LustreError: 218690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7486.949424] LustreError: 218671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840132 with bad export cookie 2389842686083493455 [ 7486.952294] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7486.968495] LustreError: 218671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7487.637744] Lustre: server umount lustre-MDT0001 complete [ 7500.659308] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 7503.837879] Key type lgssc unregistered [ 7504.206046] LNet: 223778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7504.213707] LNetError: 223778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7504.247466] LNet: Removed LNI 192.168.201.125@tcp [ 7505.296247] Key type .llcrypt unregistered [ 7505.300895] Key type ._llcrypt unregistered [ 7555.254544] Key type ._llcrypt registered [ 7555.258349] Key type .llcrypt registered [ 7555.384692] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 7556.981286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7557.031831] alg: No test for adler32 (adler32-zlib) [ 7558.115258] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 7558.450205] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 7560.144151] Key type lgssc registered [ 7561.287826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7570.482738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7570.501382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7571.719738] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7571.748944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7571.928130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7576.481163] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7585.774628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7585.843448] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7585.864475] Lustre: 225980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7586.182246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7591.590911] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7599.302114] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7606.317971] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7614.036884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7614.348207] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7614.609817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7621.285428] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7623.786063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7630.071554] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7633.269060] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7638.348508] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7646.887551] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7655.545692] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7670.125620] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7670.134407] Lustre: Skipped 1 previous similar message [ 7687.758083] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7687.766151] Lustre: Skipped 1 previous similar message [ 7724.407827] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7724.413468] Lustre: Skipped 4 previous similar messages [ 7791.342481] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7791.356237] Lustre: Skipped 7 previous similar messages [ 7813.088961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7813.097640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.110298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7813.606768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.644564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7813.647897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7817.531567] Lustre: server umount lustre-OST0000 complete [ 7821.794114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7821.804465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7821.825956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7823.841718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7823.845836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7823.877520] Lustre: Skipped 1 previous similar message [ 7827.968781] Lustre: server umount lustre-MDT0000 complete [ 7832.144528] LustreError: 226736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840477 with bad export cookie 319232043404247411 [ 7832.155963] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7832.528889] Lustre: server umount lustre-MDT0001 complete [ 7846.984346] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 7849.943234] Key type lgssc unregistered [ 7850.281751] LNet: 230793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7850.292815] LNetError: 230793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7850.318114] LNet: Removed LNI 192.168.201.125@tcp [ 7851.523514] Key type .llcrypt unregistered [ 7851.531593] Key type ._llcrypt unregistered [ 7880.911902] Key type ._llcrypt registered [ 7880.914536] Key type .llcrypt registered [ 7881.011354] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 7882.445681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7882.608319] alg: No test for adler32 (adler32-zlib) [ 7883.818736] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 7884.355609] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 7886.135224] Key type lgssc registered [ 7887.397075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7896.989597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7897.015943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7898.783414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7904.645154] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7908.322897] LustreError: 232207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7908.353204] LustreError: 232207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7913.443074] LustreError: 232206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7915.949280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7916.244333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7921.560749] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7931.249392] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7939.270748] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7948.392067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7948.987721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7956.405713] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7967.216938] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7968.508770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7989.223434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7989.227053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7989.243677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7989.249640] Lustre: Skipped 1 previous similar message [ 7994.344808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7994.354631] Lustre: Skipped 2 previous similar messages [ 7999.461030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7999.468971] Lustre: Skipped 1 previous similar message [ 8003.040061] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8003.320286] Lustre: server umount lustre-OST0000 complete [ 8008.674356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8008.678361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.687917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8009.713125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8013.762945] Lustre: server umount lustre-MDT0000 complete [ 8014.817860] LustreError: 232935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8014.856148] LustreError: 232935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8018.102834] LustreError: 232187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840663 with bad export cookie 1345991950095272321 [ 8018.116510] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8018.119054] LustreError: 232187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8018.523683] Lustre: server umount lustre-MDT0001 complete [ 8031.299859] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 8035.369839] Key type lgssc unregistered [ 8035.623388] LNet: 235900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8035.627163] LNetError: 235900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8035.652284] LNet: Removed LNI 192.168.201.125@tcp [ 8036.459354] Key type .llcrypt unregistered [ 8036.461480] Key type ._llcrypt unregistered [ 8060.975680] Key type ._llcrypt registered [ 8060.978424] Key type .llcrypt registered [ 8061.073485] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8062.072252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8062.127532] alg: No test for adler32 (adler32-zlib) [ 8063.273657] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 8063.500868] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 8065.177243] Key type lgssc registered [ 8066.241853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8338.610635] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 8342.383324] Key type lgssc unregistered [ 8342.815653] LNet: 237934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8342.826109] LNetError: 237934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8342.859294] LNet: Removed LNI 192.168.201.125@tcp [ 8343.983499] Key type .llcrypt unregistered [ 8343.985132] Key type ._llcrypt unregistered [ 8373.423667] Key type ._llcrypt registered [ 8373.438413] Key type .llcrypt registered [ 8373.567426] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 8390.332535] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8392.268729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8392.349664] alg: No test for adler32 (adler32-zlib) [ 8393.692370] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 8393.989668] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 8395.815189] Key type lgssc registered [ 8397.779890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8437.462255] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8171 sec ========== 00:18:00 (1788841080) [ 8439.937187] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:18:03 (1788841083) === [ 8444.219387] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:18:08 (1788841088) === [ 8490.107029] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 8492.997381] Key type lgssc unregistered [ 8493.284478] LNet: 243169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8493.294098] LNetError: 243169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8493.310944] LNet: Removed LNI 192.168.201.125@tcp [ 8494.271987] Key type .llcrypt unregistered [ 8494.273430] Key type ._llcrypt unregistered