[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 469460803 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003090] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008499] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010008] pid_max: default: 32768 minimum: 301 [ 0.011159] LSM: Security Framework initializing [ 0.012056] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.014050] *** VALIDATE selinux *** [ 0.022658] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027120] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028166] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030122] *** VALIDATE tmpfs *** [ 0.031493] *** VALIDATE proc *** [ 0.033133] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035225] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038020] Spectre V2 : User space: Vulnerable [ 0.040003] Speculative Store Bypass: Vulnerable [ 0.043077] debug: unmapping init [mem 0xffffffffada59000-0xffffffffada60fff] [ 0.045878] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046699] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047030] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052008] ... fixed-purpose events: 3 [ 0.052708] ... event mask: 000000070000000f [ 0.053248] rcu: Hierarchical SRCU implementation. [ 0.055161] smp: Bringing up secondary CPUs ... [ 0.056608] x86: Booting SMP configuration: [ 0.057036] .... node #0, CPUs: #1 #2 #3 [ 0.061442] smp: Brought up 1 node, 4 CPUs [ 0.062968] smpboot: Max logical packages: 1 [ 0.063029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.133906] node 0 deferred pages initialised in 66ms [ 0.136110] devtmpfs: initialized [ 0.137338] x86/mm: Memory block size: 128MB [ 0.140251] gcov: version magic: 0x41383552 [ 0.142371] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144069] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145262] pinctrl core: initialized pinctrl subsystem [ 0.146236] [ 0.146634] ************************************************************* [ 0.147011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148014] ** ** [ 0.149007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.150011] ** ** [ 0.151010] ** This means that this kernel is built to expose internal ** [ 0.152009] ** IOMMU data structures, which may compromise security on ** [ 0.153007] ** your system. ** [ 0.154008] ** ** [ 0.155007] ** If you see this message and you are not debugging the ** [ 0.156007] ** kernel, report this immediately to your vendor! ** [ 0.157009] ** ** [ 0.158010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159010] ************************************************************* [ 0.160543] NET: Registered protocol family 16 [ 0.161325] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.162036] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.163106] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.164510] cpuidle: using governor menu [ 0.165693] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.168495] PCI: Using configuration type 1 for base access [ 0.170093] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.177154] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.180034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.184597] cryptd: max_cpu_qlen set to 1000 [ 0.187304] ACPI: Added _OSI(Module Device) [ 0.189016] ACPI: Added _OSI(Processor Device) [ 0.191012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.192008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.195623] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.201397] ACPI: Interpreter enabled [ 0.202049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203008] ACPI: Using IOAPIC for interrupt routing [ 0.205074] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.207351] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.217241] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.219052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.222010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.224096] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.228176] acpiphp: Slot [2] registered [ 0.229091] acpiphp: Slot [5] registered [ 0.230097] acpiphp: Slot [6] registered [ 0.232068] acpiphp: Slot [7] registered [ 0.233074] acpiphp: Slot [8] registered [ 0.234082] acpiphp: Slot [9] registered [ 0.235115] acpiphp: Slot [10] registered [ 0.237095] acpiphp: Slot [3] registered [ 0.238081] acpiphp: Slot [4] registered [ 0.239074] acpiphp: Slot [11] registered [ 0.240073] acpiphp: Slot [12] registered [ 0.242064] acpiphp: Slot [13] registered [ 0.243232] acpiphp: Slot [14] registered [ 0.244097] acpiphp: Slot [15] registered [ 0.245092] acpiphp: Slot [16] registered [ 0.246058] acpiphp: Slot [17] registered [ 0.248074] acpiphp: Slot [18] registered [ 0.249101] acpiphp: Slot [19] registered [ 0.250079] acpiphp: Slot [20] registered [ 0.251088] acpiphp: Slot [21] registered [ 0.252089] acpiphp: Slot [22] registered [ 0.253094] acpiphp: Slot [23] registered [ 0.255096] acpiphp: Slot [24] registered [ 0.256086] acpiphp: Slot [25] registered [ 0.257103] acpiphp: Slot [26] registered [ 0.259117] acpiphp: Slot [27] registered [ 0.260075] acpiphp: Slot [28] registered [ 0.262113] acpiphp: Slot [29] registered [ 0.264146] acpiphp: Slot [30] registered [ 0.266086] acpiphp: Slot [31] registered [ 0.267078] PCI host bridge to bus 0000:00 [ 0.268022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.270019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.273025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.275019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.277041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.279021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.281170] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.284550] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.287239] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.296712] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.302599] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.306019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.309015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.311013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.313548] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.316068] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.319045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.321783] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.327015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.339990] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.346023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.352250] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.370015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.378021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.397022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.411011] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.421014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.430023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.447018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.456027] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.465017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.471017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.489019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.498551] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.504016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.510018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.526020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.535053] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.541016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.548015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.567017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.579713] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.593016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.606024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.628021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.642380] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.644481] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.647501] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.651480] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.653318] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.658250] iommu: Default domain type: Passthrough [ 0.660440] SCSI subsystem initialized [ 0.662186] ACPI: bus type USB registered [ 0.663191] usbcore: registered new interface driver usbfs [ 0.666179] usbcore: registered new interface driver hub [ 0.668134] usbcore: registered new device driver usb [ 0.670262] pps_core: LinuxPPS API ver. 1 registered [ 0.672024] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.676111] PTP clock support registered [ 0.678119] EDAC MC: Ver: 3.0.0 [ 0.679436] PCI: Using ACPI for IRQ routing [ 0.680977] NetLabel: Initializing [ 0.681011] NetLabel: domain hash size = 128 [ 0.682011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.683101] NetLabel: unlabeled traffic allowed by default [ 0.686108] vgaarb: loaded [ 0.687336] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.689019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.693006] clocksource: Switched to clocksource kvm-clock [ 0.809197] VFS: Disk quotas dquot_6.6.0 [ 0.811537] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.814259] *** VALIDATE ramfs *** [ 0.815460] *** VALIDATE hugetlbfs *** [ 0.816844] pnp: PnP ACPI init [ 0.819281] pnp: PnP ACPI: found 6 devices [ 0.850445] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.853780] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.855986] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.858028] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.860212] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.862852] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.865621] NET: Registered protocol family 2 [ 0.867975] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.873086] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.876650] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.882295] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.886053] TCP: Hash tables configured (established 65536 bind 65536) [ 0.889086] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.892374] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.895530] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.899050] NET: Registered protocol family 1 [ 0.901846] RPC: Registered named UNIX socket transport module. [ 0.903898] RPC: Registered udp transport module. [ 0.905636] RPC: Registered tcp transport module. [ 0.907358] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.909578] NET: Registered protocol family 44 [ 0.911456] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.914133] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.916682] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.918890] PCI: CLS 0 bytes, default 64 [ 0.921223] Unpacking initramfs... [ 2.346935] debug: unmapping init [mem 0xffff8ccdfcc54000-0xffff8ccdfffbffff] [ 2.351307] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.353665] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.356956] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.900735] Initialise system trusted keyrings [ 2.902537] Key type blacklist registered [ 2.906186] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.917068] zbud: loaded [ 2.922111] *** VALIDATE nfs *** [ 2.923538] *** VALIDATE nfs4 *** [ 2.925214] pstore: using deflate compression [ 2.929197] Platform Keyring initialized [ 3.063247] NET: Registered protocol family 38 [ 3.064390] Key type asymmetric registered [ 3.065765] Asymmetric key parser 'x509' registered [ 3.067469] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.070103] io scheduler mq-deadline registered [ 3.070987] io scheduler kyber registered [ 3.071877] io scheduler bfq registered [ 3.073149] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.075273] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.077390] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.080037] ACPI: Power Button [PWRF] [ 3.087153] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.094926] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.111351] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.120652] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.132495] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.158786] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.186281] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.191395] Non-volatile memory driver v1.3 [ 3.193090] Linux agpgart interface v0.103 [ 3.229069] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.232246] vda: detected capacity change from 0 to 75100160 [ 3.249529] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.252595] vdb: detected capacity change from 0 to 1073741824 [ 3.267080] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.269779] vdc: detected capacity change from 0 to 2621440000 [ 3.282842] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.285746] vdd: detected capacity change from 0 to 2621440000 [ 3.303300] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.306309] vde: detected capacity change from 0 to 4294967296 [ 3.323104] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.326289] vdf: detected capacity change from 0 to 4294967296 [ 3.334931] libphy: Fixed MDIO Bus: probed [ 3.342786] usbcore: registered new interface driver usbserial_generic [ 3.345508] usbserial: USB Serial support registered for generic [ 3.348043] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.353284] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.355377] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.357971] mousedev: PS/2 mouse device common for all mice [ 3.361306] rtc_cmos 00:05: RTC can wake from S4 [ 3.364233] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.364523] rtc_cmos 00:05: registered as rtc0 [ 3.372881] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.376183] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.377159] intel_pstate: CPU model not supported [ 3.383688] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.388123] hid: raw HID events driver (C) Jiri Kosina [ 3.390063] usbcore: registered new interface driver usbhid [ 3.392237] usbhid: USB HID core driver [ 3.393908] drop_monitor: Initializing network drop monitor service [ 3.396647] Initializing XFRM netlink socket [ 3.398743] NET: Registered protocol family 10 [ 3.401654] Segment Routing with IPv6 [ 3.403152] NET: Registered protocol family 17 [ 3.405453] mpls_gso: MPLS GSO support [ 3.411961] RAS: Correctable Errors collector initialized. [ 3.413593] AVX version of gcm_enc/dec engaged. [ 3.414972] AES CTR mode by8 optimization enabled [ 3.495728] sched_clock: Marking stable (3495704068, 0)->(4403131256, -907427188) [ 3.499221] registered taskstats version 1 [ 3.502187] Loading compiled-in X.509 certificates [ 3.504329] zswap: loaded using pool lzo/zbud [ 3.529066] Key type big_key registered [ 3.541816] Key type encrypted registered [ 3.543593] ima: No TPM chip found, activating TPM-bypass! [ 3.545084] ima: Allocated hash algorithm: sha1 [ 3.546254] ima: No architecture policies found [ 3.548097] evm: Initialising EVM extended attributes: [ 3.549376] evm: security.selinux [ 3.550276] evm: security.ima [ 3.551113] evm: security.capability [ 3.552022] evm: HMAC attrs: 0x1 [ 3.553942] rtc_cmos 00:05: setting system clock to 2026-08-27 10:21:55 UTC (1787826115) [ 3.558177] debug: unmapping init [mem 0xffffffffaea03000-0xffffffffaebfffff] [ 3.560314] debug: unmapping init [mem 0xffffffffad782000-0xffffffffada58fff] [ 3.567091] Write protecting the kernel read-only data: 28672k [ 3.571379] debug: unmapping init [mem 0xffffffffabe03000-0xffffffffabffffff] [ 3.574842] debug: unmapping init [mem 0xffffffffac714000-0xffffffffac7fffff] [ 3.606238] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.613907] systemd[1]: Detected virtualization kvm. [ 3.615424] systemd[1]: Detected architecture x86-64. [ 3.617247] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.642722] systemd[1]: No hostname configured. [ 3.644090] systemd[1]: Set hostname to . [ 3.646298] random: systemd: uninitialized urandom read (16 bytes read) [ 3.648764] systemd[1]: Initializing machine ID from random generator. [ 3.699996] random: ln: uninitialized urandom read (6 bytes read) [ 3.778151] random: systemd: uninitialized urandom read (16 bytes read) [ 3.780849] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.786449] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.790726] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.310622] device-mapper: uevent: version 1.0.3 [ 4.313074] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 4.966410] virtio_net virtio0 ens2: renamed from eth0 [ 4.982122] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.051652] scsi host0: ata_piix [ 5.138276] scsi host1: ata_piix [ 5.140026] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.142319] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.731388] dracut-initqueue[576]: RTNETLINK answers: File exists [ 10.059906] random: crng init done [ 10.061450] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.369783] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.566830] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.809364] SELinux: Disabled at runtime. [ 11.874886] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.887478] systemd[1]: Detected virtualization kvm. [ 11.890388] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.403706] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.407075] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.411906] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.416368] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.419721] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.428462] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.433154] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ 12.647894] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.045514] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.317317] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.337844] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.451995] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.467992] EDAC sbridge: Ver: 1.1.2 [ 14.766685] Key type dns_resolver registered [ 15.071763] NFS: Registering the id_resolver key type [ 15.073045] Key type id_resolver registered [ 15.074358] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg112-server login: [ 43.166411] libcfs: loading out-of-tree module taints kernel. [ 43.185629] Key type ._llcrypt registered [ 43.187269] Key type .llcrypt registered [ 43.226154] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 52.317643] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 52.934269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.940761] alg: No test for adler32 (adler32-zlib) [ 53.972406] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 54.316057] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 55.951168] Key type lgssc registered [ 57.646397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.191079] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 113.089596] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 127.777976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 127.823253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 129.288751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 129.320448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 129.382710] Lustre: lustre-MDT0000: new disk, initializing [ 129.521366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 129.558936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 134.271788] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 149.163301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.332709] Lustre: 6478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 149.413143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 149.422090] Lustre: Skipped 1 previous similar message [ 149.619782] Lustre: lustre-MDT0001: new disk, initializing [ 149.721380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.752236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 149.760641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 155.137552] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.262036] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 170.824690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 171.197812] Lustre: lustre-OST0000: new disk, initializing [ 171.206242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 171.223404] Lustre: 8415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 171.306274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.765074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 174.775620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 174.883713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 179.194484] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.696112] hrtimer: interrupt took 2332881 ns [ 196.580486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.707796] Lustre: lustre-OST0001: new disk, initializing [ 196.711556] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 196.715961] Lustre: 9488:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.814820] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.299094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 203.309866] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 203.379207] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 204.337376] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.927803] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 222.786941] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 229.239416] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing check_logdir /tmp/testlogs/ [ 234.488128] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing yml_node [ 238.992098] Lustre: DEBUG MARKER: Client: 2.17.57.68 [ 241.792596] Lustre: DEBUG MARKER: MDS: 2.17.57.68 [ 245.050722] Lustre: DEBUG MARKER: OSS: 2.17.57.68 [ 246.760646] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 06:25:57 EDT 2026 [ 268.660761] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 270.961597] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 288.226936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 288.238374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.253718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.274211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.278427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.300224] Lustre: Skipped 1 previous similar message [ 292.741243] Lustre: server umount lustre-MDT0000 complete [ 297.818256] LustreError: 9487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826409 with bad export cookie 15406380443465826264 [ 297.824786] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.827460] LustreError: 9487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 300.515483] LustreError: 6483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.516917] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.547055] LustreError: 6483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 300.590594] Lustre: Skipped 2 previous similar messages [ 300.621046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.636849] Lustre: Skipped 2 previous similar messages [ 302.569383] LustreError: 6484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.595981] LustreError: 6484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 307.683604] LustreError: 8988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.684379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.706887] Lustre: Skipped 3 previous similar messages [ 312.287107] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 312.562720] Lustre: server umount lustre-MDT0001 complete [ 317.647748] Lustre: server umount lustre-OST0000 complete [ 321.233984] Lustre: server umount lustre-OST0001 complete [ 330.070930] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 342.971568] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 386.063161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.464505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 386.526856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 386.625722] Lustre: lustre-MDT0000: new disk, initializing [ 386.740199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.786539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 392.105886] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 404.117348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.416791] Lustre: 15042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 404.554744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 404.558762] Lustre: Skipped 1 previous similar message [ 404.709481] Lustre: lustre-MDT0001: new disk, initializing [ 404.880204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 404.983628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 404.998211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 409.505672] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 420.279755] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 426.703396] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 434.044531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 434.354542] Lustre: lustre-OST0000: new disk, initializing [ 434.361671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 434.368450] Lustre: 17011:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 434.470672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 436.069711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 436.087722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 436.239579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 441.301395] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 453.136655] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 461.357390] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 461.832665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 468.840694] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 469.295219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 477.163327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.167460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 477.186545] Lustre: Skipped 2 previous similar messages [ 486.879237] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 487.278616] Lustre: server umount lustre-OST0000 complete [ 492.011502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 492.027134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.050402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 492.056149] Lustre: Skipped 3 previous similar messages [ 497.902853] Lustre: server umount lustre-MDT0000 complete [ 502.758158] LustreError: 15054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.774520] LustreError: 15054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 503.223752] LustreError: 15034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826615 with bad export cookie 15406380443465827797 [ 503.246152] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 503.246787] LustreError: 15034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 503.793141] Lustre: server umount lustre-MDT0001 complete [ 508.385350] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 06:30:18 (1787826618) [ 516.648400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.321188] LustreError: 19278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.412467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 522.844505] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 527.855298] LustreError: 19279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.884635] LustreError: 19279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 536.899055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 543.699336] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 553.497781] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 561.394056] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 570.015484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 580.668241] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 591.997638] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 602.594508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.613918] Lustre: Skipped 1 previous similar message [ 602.625166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 602.642725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 602.653753] Lustre: Skipped 4 previous similar messages [ 607.960374] Lustre: server umount lustre-OST0000 complete [ 619.039337] Lustre: server umount lustre-MDT0000 complete [ 622.748269] LustreError: 19258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826734 with bad export cookie 15406380443465828679 [ 622.757140] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 622.762991] LustreError: 19258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 623.248576] Lustre: server umount lustre-MDT0001 complete [ 638.988355] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 642.339860] Key type lgssc unregistered [ 642.658391] LNet: 22815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.669419] LNetError: 22815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 642.688186] LNet: Removed LNI 192.168.201.112@tcp [ 643.511280] Key type .llcrypt unregistered [ 643.515079] Key type ._llcrypt unregistered [ 668.022594] Key type ._llcrypt registered [ 668.026297] Key type .llcrypt registered [ 668.220178] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 669.603027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 669.790106] alg: No test for adler32 (adler32-zlib) [ 671.574317] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 672.097163] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 673.872086] Key type lgssc registered [ 675.372791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 687.490924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 687.517723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.196944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 695.770385] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 699.368883] LustreError: 24231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 699.391124] LustreError: 24231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 704.492278] LustreError: 24232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.587876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 708.184476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 713.920436] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 723.477986] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 729.334625] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 736.019619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 736.404773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 744.225899] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 752.442075] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 759.913030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 770.542479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 770.571579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 770.577499] Lustre: Skipped 1 previous similar message [ 770.612757] Lustre: Skipped 1 previous similar message [ 775.648995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 775.660303] Lustre: Skipped 1 previous similar message [ 776.268049] Lustre: server umount lustre-OST0000 complete [ 784.867169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 784.884728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 784.900163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 785.889174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.008537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 791.020346] Lustre: Skipped 3 previous similar messages [ 796.127598] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 796.131452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 796.150743] Lustre: Skipped 1 previous similar message [ 796.336539] Lustre: server umount lustre-MDT0000 complete [ 800.225361] LustreError: 26529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 800.240634] LustreError: 26529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 800.624686] LustreError: 24213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826912 with bad export cookie 3668201743751680828 [ 800.626334] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 800.642249] LustreError: 24213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 801.310708] Lustre: server umount lustre-MDT0001 complete [ 813.237779] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 817.739490] Key type lgssc unregistered [ 818.201755] LNet: 28211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.207687] LNetError: 28211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.217964] LNet: Removed LNI 192.168.201.112@tcp [ 819.229444] Key type .llcrypt unregistered [ 819.231891] Key type ._llcrypt unregistered [ 850.872327] Key type ._llcrypt registered [ 850.875639] Key type .llcrypt registered [ 851.032118] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 852.629892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 852.657169] alg: No test for adler32 (adler32-zlib) [ 853.711640] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 854.012934] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 855.712096] Key type lgssc registered [ 857.081200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 867.275629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 867.305107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.129677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 874.441220] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 879.589573] LustreError: 29624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.636764] LustreError: 29624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 884.726251] LustreError: 29625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.445899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.955423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 893.714922] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 901.554695] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 907.617795] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 923.620197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 924.097454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 931.956991] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 939.470648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 940.640626] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 947.681134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 947.690618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.708438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 949.732364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 949.745601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 949.756675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.994524] Lustre: server umount lustre-OST0000 complete [ 959.458796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 959.466949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 959.483966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 959.497539] Lustre: Skipped 1 previous similar message [ 964.336259] Lustre: server umount lustre-MDT0000 complete [ 965.091635] LustreError: 29620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.108685] LustreError: 29620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 969.251283] LustreError: 29606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827081 with bad export cookie 5963849701582379565 [ 969.265792] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 969.266499] LustreError: 29606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 969.744380] Lustre: server umount lustre-MDT0001 complete [ 982.094763] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 986.976758] Key type lgssc unregistered [ 987.570522] LNet: 33509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.579099] LNetError: 33509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.590315] LNet: Removed LNI 192.168.201.112@tcp [ 988.746540] Key type .llcrypt unregistered [ 988.748959] Key type ._llcrypt unregistered [ 1019.369626] Key type ._llcrypt registered [ 1019.375119] Key type .llcrypt registered [ 1019.605507] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1022.315300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1022.371254] alg: No test for adler32 (adler32-zlib) [ 1023.667455] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1024.079773] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1025.816423] Key type lgssc registered [ 1027.239869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1035.474724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1035.529515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1037.136251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.980533] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1047.524093] LustreError: 34926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1047.568936] LustreError: 34926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1052.294730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.641543] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1052.919193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1058.642871] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1067.146474] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1074.957819] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1084.667298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1085.071800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.182903] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1102.725733] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1107.685941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1118.177904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1118.184020] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.189751] LustreError: Skipped 1 previous similar message [ 1118.192250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1118.207149] Lustre: Skipped 1 previous similar message [ 1118.226116] Lustre: Skipped 1 previous similar message [ 1120.101750] Lustre: server umount lustre-OST0000 complete [ 1128.417452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.419507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1128.430487] Lustre: Skipped 1 previous similar message [ 1130.778381] Lustre: server umount lustre-MDT0000 complete [ 1133.538349] LustreError: 34926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.559311] LustreError: 34926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1135.100251] LustreError: 34906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827247 with bad export cookie 2416698746035893982 [ 1135.107408] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.455632] Lustre: server umount lustre-MDT0001 complete [ 1148.679815] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1152.206940] Key type lgssc unregistered [ 1152.636842] LNet: 38489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.653632] LNetError: 38489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.665442] LNet: Removed LNI 192.168.201.112@tcp [ 1153.847339] Key type .llcrypt unregistered [ 1153.849175] Key type ._llcrypt unregistered [ 1178.124419] Key type ._llcrypt registered [ 1178.126828] Key type .llcrypt registered [ 1178.257765] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1179.719690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1179.768095] alg: No test for adler32 (adler32-zlib) [ 1180.930422] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1181.215496] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1182.911925] Key type lgssc registered [ 1184.523671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1194.161858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1194.210908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.046839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1201.012411] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1205.230203] LustreError: 39902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.258691] LustreError: 39902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1210.336945] LustreError: 39903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.348255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1212.874710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1218.000090] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1226.758351] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1234.873673] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1244.636554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1245.278128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1254.606566] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1264.779570] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1265.748208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1272.800700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1272.806634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1272.834626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1275.878583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1275.879626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.890044] Lustre: Skipped 1 previous similar message [ 1278.147606] Lustre: lustre-OST0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 1278.162052] Lustre: Skipped 1 previous similar message [ 1278.522777] Lustre: server umount lustre-OST0000 complete [ 1299.966643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1299.992991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.023674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1306.598966] LustreError: 40887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.629475] LustreError: 40887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1306.767175] Lustre: server umount lustre-MDT0000 complete [ 1311.338909] LustreError: 39882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827423 with bad export cookie 17856093919106138658 [ 1311.347038] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1311.352511] LustreError: 39882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1317.864783] Lustre: server umount lustre-MDT0001 complete [ 1329.010433] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1331.749335] Key type lgssc unregistered [ 1332.077832] LNet: 43616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.089348] LNetError: 43616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1333.161377] LNet: Removed LNI 192.168.201.112@tcp [ 1334.145626] Key type .llcrypt unregistered [ 1334.149082] Key type ._llcrypt unregistered [ 1359.740914] Key type ._llcrypt registered [ 1359.742627] Key type .llcrypt registered [ 1359.827507] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1361.150150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1361.199483] alg: No test for adler32 (adler32-zlib) [ 1362.292478] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1362.590133] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1364.290401] Key type lgssc registered [ 1365.321790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1372.537054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1372.561748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.038268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.708507] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1383.396898] LustreError: 45034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.429765] LustreError: 45034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1387.886763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.262848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1393.014775] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1402.146044] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1409.813203] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1418.907087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1419.631220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1426.528628] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1437.785751] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1443.288825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1449.187192] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 1449.955515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1449.967669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.981863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1450.996551] Lustre: server umount lustre-MDT0000 complete [ 1453.543907] LustreError: 45035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1453.560733] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1454.757367] LustreError: 45016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827566 with bad export cookie 15146075929045979794 [ 1454.766310] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1454.773971] LustreError: 45016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1458.659479] LustreError: 45030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1458.663784] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1458.679236] LustreError: 45030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1458.726507] Lustre: Skipped 1 previous similar message [ 1458.731939] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1461.431804] Lustre: server umount lustre-MDT0001 complete [ 1483.192296] Lustre: server umount lustre-OST0000 complete [ 1504.251375] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1506.780518] Key type lgssc unregistered [ 1507.026889] LNet: 48901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1507.031790] LNetError: 48901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.069767] LNet: Removed LNI 192.168.201.112@tcp [ 1508.803633] Key type .llcrypt unregistered [ 1508.805665] Key type ._llcrypt unregistered [ 1532.995439] Key type ._llcrypt registered [ 1532.999307] Key type .llcrypt registered [ 1533.100454] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1534.187535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1534.222538] alg: No test for adler32 (adler32-zlib) [ 1535.333424] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1535.607095] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1537.335403] Key type lgssc registered [ 1539.281353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1547.640207] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1547.684244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1563.423521] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1568.543755] LustreError: 50288:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [ 1595.231502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1601.155842] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1609.465371] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1778.079893] Lustre: server umount lustre-OST0000 complete [ 1799.749927] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1803.622501] Key type lgssc unregistered [ 1803.976281] LNet: 51967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.985492] LNetError: 51967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.008704] LNet: Removed LNI 192.168.201.112@tcp [ 1805.152036] Key type .llcrypt unregistered [ 1805.154828] Key type ._llcrypt unregistered [ 1830.948266] Key type ._llcrypt registered [ 1830.949956] Key type .llcrypt registered [ 1831.073716] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1832.181236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1832.204800] alg: No test for adler32 (adler32-zlib) [ 1833.314597] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1833.562221] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1835.319181] Key type lgssc registered [ 1836.207383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1844.283772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1844.299062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.809947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1851.080661] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1855.985791] LustreError: 53388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1856.006534] LustreError: 53388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1859.972676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.419738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1864.837489] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1871.564064] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1877.034837] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1883.019304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1883.367660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.187632] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1890.553949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1895.773490] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1901.035869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1901.037787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.044393] LustreError: Skipped 1 previous similar message [ 1901.045832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1901.059770] Lustre: Skipped 1 previous similar message [ 1906.147513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1906.158847] Lustre: Skipped 2 previous similar messages [ 1911.272654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1911.276553] Lustre: Skipped 1 previous similar message [ 1912.287153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1912.610179] Lustre: server umount lustre-OST0000 complete [ 1916.385572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.389966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1916.403679] Lustre: Skipped 1 previous similar message [ 1921.506832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.509772] Lustre: Skipped 2 previous similar messages [ 1922.891061] Lustre: server umount lustre-MDT0000 complete [ 1926.628540] LustreError: 53388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1926.659413] LustreError: 53388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1926.684576] LustreError: 53416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828038 with bad export cookie 3467150845856257964 [ 1926.690351] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1926.691032] LustreError: 53416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1927.131707] Lustre: server umount lustre-MDT0001 complete [ 1936.607437] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1938.889845] Key type lgssc unregistered [ 1939.118762] LNet: 56927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.126582] LNetError: 56927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1940.137447] LNet: Removed LNI 192.168.201.112@tcp [ 1940.829142] Key type .llcrypt unregistered [ 1940.830914] Key type ._llcrypt unregistered [ 1963.070418] Key type ._llcrypt registered [ 1963.073501] Key type .llcrypt registered [ 1963.169896] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1964.035365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.060799] alg: No test for adler32 (adler32-zlib) [ 1965.184786] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1965.418091] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1967.071462] Key type lgssc registered [ 1968.401971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1976.541893] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1976.555053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1992.095643] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1997.215699] LustreError: 58293:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [ 2023.903688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.694710] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2035.672932] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2042.007904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.394640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.434111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2046.501481] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2047.461924] LustreError: 58318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.475322] LustreError: 58318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2052.577166] LustreError: 58319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.415994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.718180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2059.534737] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2066.909575] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2072.944067] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2075.105063] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2075.119239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2075.134143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2078.182130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2078.190425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2078.222055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2081.248342] Lustre: server umount lustre-OST0000 complete [ 2109.928791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.952396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2109.957594] Lustre: Skipped 3 previous similar messages [ 2114.060607] Lustre: server umount lustre-MDT0000 complete [ 2115.054091] LustreError: 59333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.078883] LustreError: 59333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2118.520248] LustreError: 58300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828230 with bad export cookie 2458096242479338567 [ 2118.531826] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2118.538268] LustreError: 58300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2125.261455] Lustre: server umount lustre-MDT0001 complete [ 2138.078454] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2141.235554] Key type lgssc unregistered [ 2141.557046] LNet: 62039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2141.569940] LNetError: 62039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2142.647405] LNet: Removed LNI 192.168.201.112@tcp [ 2143.912851] Key type .llcrypt unregistered [ 2143.918454] Key type ._llcrypt unregistered [ 2169.799230] Key type ._llcrypt registered [ 2169.803282] Key type .llcrypt registered [ 2169.958414] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2171.506066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2171.680306] alg: No test for adler32 (adler32-zlib) [ 2172.730060] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2172.916296] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2174.575784] Key type lgssc registered [ 2175.906220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2184.126541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2184.161079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.879229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2190.317989] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2195.429698] LustreError: 63454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.450117] LustreError: 63454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2200.555123] LustreError: 63453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.639074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.005876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2206.491183] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2214.781569] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2220.720656] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2227.796686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2228.130626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.192431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2235.497132] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2243.264103] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2266.081297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2266.082883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.095460] LustreError: Skipped 1 previous similar message [ 2266.096207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2266.168415] Lustre: Skipped 1 previous similar message [ 2271.208422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2271.224955] Lustre: Skipped 2 previous similar messages [ 2276.321465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2276.326440] Lustre: Skipped 1 previous similar message [ 2276.327982] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 2276.423852] Lustre: server umount lustre-OST0000 complete [ 2281.453173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.465674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2281.477596] Lustre: Skipped 1 previous similar message [ 2281.513518] Lustre: Skipped 1 previous similar message [ 2286.562444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2286.568060] Lustre: Skipped 1 previous similar message [ 2295.263807] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2295.495781] Lustre: server umount lustre-MDT0000 complete [ 2296.806339] LustreError: 63454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.823721] LustreError: 63454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2299.603419] LustreError: 63435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828411 with bad export cookie 11191518627078008698 [ 2299.612702] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2300.012787] Lustre: server umount lustre-MDT0001 complete [ 2312.247527] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2315.968295] Key type lgssc unregistered [ 2316.332592] LNet: 67012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2316.349523] LNetError: 67012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2316.376930] LNet: Removed LNI 192.168.201.112@tcp [ 2317.993208] Key type .llcrypt unregistered [ 2317.998308] Key type ._llcrypt unregistered [ 2350.557053] Key type ._llcrypt registered [ 2350.560703] Key type .llcrypt registered [ 2350.736100] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2352.440868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2352.486205] alg: No test for adler32 (adler32-zlib) [ 2353.744579] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2354.183721] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2355.951666] Key type lgssc registered [ 2357.218410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2367.584388] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 07:01:17 (1787828477) [ 2374.772314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2374.819478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.455553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2380.686099] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2386.913876] LustreError: 68732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.930682] LustreError: 68732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2389.765392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2390.085802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2394.970766] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2403.646560] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2409.509924] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2416.017113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2416.396842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2421.557179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2422.758948] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2431.026271] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2439.778412] Lustre: Failing over lustre-MDT0000 [ 2440.059390] Lustre: server umount lustre-MDT0000 complete [ 2440.163239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2440.181580] LustreError: 69972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.192646] Lustre: Skipped 2 previous similar messages [ 2440.237650] LustreError: 69972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2442.991815] LustreError: 69972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.283959] LustreError: 68732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.310965] LustreError: 68732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2449.756283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2449.893151] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2450.072826] Lustre: *** cfs_fail_loc=135, val=0*** [ 2450.196775] LustreError: 71821:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2450.205862] LustreError: 71821:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.112@tcp: cfg command failed: rc = -2 [ 2450.215433] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2450.215433] [ 2450.238588] LustreError: MGC192.168.201.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2450.260378] LustreError: 71809:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2450.287665] LustreError: 71809:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2450.303264] LustreError: 71809:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2450.312064] LustreError: 71809:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2450.364389] Lustre: server umount lustre-MDT0000 complete [ 2450.370926] LustreError: 71809:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2450.404719] LustreError: 69459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.418601] LustreError: 69459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2457.161113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.320616] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2457.329165] LustreError: 68723:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2457.711444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2457.826288] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2458.297785] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2462.273373] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2462.716776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2462.754234] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2462.818241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2470.880301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2470.890185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.914736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2472.930536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2472.932043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.220811] Lustre: server umount lustre-OST0000 complete [ 2483.172109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2483.173528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2483.187181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2483.209601] Lustre: Skipped 1 previous similar message [ 2483.212959] Lustre: Skipped 1 previous similar message [ 2486.548325] Lustre: server umount lustre-MDT0000 complete [ 2488.302087] LustreError: 69972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.323677] LustreError: 69972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2490.831691] LustreError: 68713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828602 with bad export cookie 17415113664646406376 [ 2490.849152] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2491.488031] Lustre: server umount lustre-MDT0001 complete [ 2508.057700] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2512.310245] Key type lgssc unregistered [ 2512.751566] LNet: 73604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2512.760548] LNetError: 73604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2512.776990] LNet: Removed LNI 192.168.201.112@tcp [ 2514.413947] Key type .llcrypt unregistered [ 2514.416735] Key type ._llcrypt unregistered [ 2544.247239] Key type ._llcrypt registered [ 2544.249492] Key type .llcrypt registered [ 2544.334155] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2545.912494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2545.998918] alg: No test for adler32 (adler32-zlib) [ 2547.053527] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2547.360766] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2549.119577] Key type lgssc registered [ 2550.414709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2558.508188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2558.520870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.947121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2564.556816] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2569.199276] LustreError: 75019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.229054] LustreError: 75019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2573.484618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2573.900102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2578.410687] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2585.958146] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2592.320312] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2600.450846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2600.921405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.044513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2607.663645] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2615.947265] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2622.679985] Lustre: Failing over lustre-MDT0000 [ 2622.902573] Lustre: server umount lustre-MDT0000 complete [ 2623.462822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2623.478520] Lustre: Skipped 2 previous similar messages [ 2623.483055] LustreError: 75019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.492584] LustreError: 75019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2628.582143] LustreError: 75020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.597931] LustreError: 75020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2633.118052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.213664] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2633.356804] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2633.494925] LustreError: 78105:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2633.506722] LustreError: 78105:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.112@tcp: cfg command failed: rc = -9 [ 2633.517871] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2633.517871] [ 2633.531254] LustreError: MGC192.168.201.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2633.551126] LustreError: 78094:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2633.560323] LustreError: 78094:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2633.577553] LustreError: 78094:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2633.596318] LustreError: 78094:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2633.626474] Lustre: server umount lustre-MDT0000 complete [ 2633.634898] LustreError: 78094:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2633.697303] LustreError: 76245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.717251] LustreError: 76245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2638.821888] LustreError: 75019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.837250] LustreError: 75019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2640.382924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.524469] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2640.865079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2640.932562] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2644.422140] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2646.007717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2646.046839] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2646.129384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2646.476583] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2656.246411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2656.256915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2656.272862] Lustre: Skipped 1 previous similar message [ 2661.363638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2661.382826] Lustre: Skipped 2 previous similar messages [ 2661.844420] Lustre: server umount lustre-OST0000 complete [ 2666.468581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2666.470507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2666.486820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2666.508021] Lustre: Skipped 1 previous similar message [ 2671.585108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2671.590764] Lustre: Skipped 2 previous similar messages [ 2676.705187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2676.724290] Lustre: Skipped 1 previous similar message [ 2680.287248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2680.750483] Lustre: server umount lustre-MDT0000 complete [ 2681.827947] LustreError: 75020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.854069] LustreError: 75020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2684.572463] LustreError: 75001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828796 with bad export cookie 2008019995377340298 [ 2684.582882] LustreError: 75001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2684.589637] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2685.002421] Lustre: server umount lustre-MDT0001 complete [ 2697.221177] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2700.520467] Key type lgssc unregistered [ 2700.792559] LNet: 79892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2700.805948] LNetError: 79892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2700.830589] LNet: Removed LNI 192.168.201.112@tcp [ 2701.873867] Key type .llcrypt unregistered [ 2701.886626] Key type ._llcrypt unregistered [ 2716.353255] Key type ._llcrypt registered [ 2716.354935] Key type .llcrypt registered [ 2716.497492] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2716.820637] Key type .llcrypt unregistered [ 2716.828384] Key type ._llcrypt unregistered [ 2743.111854] Key type ._llcrypt registered [ 2743.119083] Key type .llcrypt registered [ 2743.256587] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2744.775053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2744.924332] alg: No test for adler32 (adler32-zlib) [ 2746.238972] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2746.532640] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2748.301539] Key type lgssc registered [ 2749.960394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2760.133248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2760.172663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.023279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2769.402294] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2771.429463] LustreError: 81797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.467355] LustreError: 81797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2776.559282] LustreError: 81798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.897178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2780.410967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2785.612377] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2793.577657] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2800.259919] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2809.834232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2810.546551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.224456] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2828.539402] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2832.058386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2846.699138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2846.712682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.742389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2847.714212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2847.715252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.721894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2850.404273] Lustre: server umount lustre-OST0000 complete [ 2857.450627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2857.460252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.483562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2857.485946] Lustre: Skipped 1 previous similar message [ 2863.076990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2863.084332] Lustre: Skipped 2 previous similar messages [ 2868.191922] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2868.193135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.213796] Lustre: Skipped 1 previous similar message [ 2868.356337] Lustre: server umount lustre-MDT0000 complete [ 2872.293503] LustreError: 81798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2872.329057] LustreError: 81798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2873.132363] LustreError: 81779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828985 with bad export cookie 8408658333876091077 [ 2873.141267] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2873.148903] LustreError: 81779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2873.542964] Lustre: server umount lustre-MDT0001 complete [ 2886.084516] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2889.204807] Key type lgssc unregistered [ 2889.477559] LNet: 85360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2889.490227] LNetError: 85360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2890.534584] LNet: Removed LNI 192.168.201.112@tcp [ 2891.404838] Key type .llcrypt unregistered [ 2891.406628] Key type ._llcrypt unregistered [ 2915.918310] Key type ._llcrypt registered [ 2915.921963] Key type .llcrypt registered [ 2916.065652] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2917.405020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2917.447524] alg: No test for adler32 (adler32-zlib) [ 2918.460676] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2918.714242] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2920.407518] Key type lgssc registered [ 2921.958826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2930.535785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2930.574679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.061781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.169837] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2941.478847] LustreError: 86773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2941.502735] LustreError: 86773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2946.266911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2946.529773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2946.705192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2952.083669] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2960.645548] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2967.569827] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2976.299845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2976.778811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2981.890842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2984.905958] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2995.897194] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3006.436572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.466275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3006.471618] Lustre: Skipped 1 previous similar message [ 3009.720366] Lustre: server umount lustre-OST0000 complete [ 3013.089225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3013.094939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3013.106766] Lustre: Skipped 1 previous similar message [ 3013.111788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3016.676982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3016.679151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3016.687665] Lustre: Skipped 1 previous similar message [ 3019.419980] Lustre: server umount lustre-MDT0000 complete [ 3021.810749] LustreError: 87500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3021.829046] LustreError: 87500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3023.556797] LustreError: 86753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829135 with bad export cookie 8418535683023500737 [ 3023.560309] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3023.570843] LustreError: 86753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3023.936477] Lustre: server umount lustre-MDT0001 complete [ 3034.431533] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3037.206463] Key type lgssc unregistered [ 3037.511475] LNet: 90309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3037.522992] LNetError: 90309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3037.548333] LNet: Removed LNI 192.168.201.112@tcp [ 3038.392744] Key type .llcrypt unregistered [ 3038.394786] Key type ._llcrypt unregistered [ 3059.867934] Key type ._llcrypt registered [ 3059.870378] Key type .llcrypt registered [ 3059.943110] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3061.077927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3061.108130] alg: No test for adler32 (adler32-zlib) [ 3062.092484] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 3062.308312] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3063.959949] Key type lgssc registered [ 3064.985620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3072.856769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3072.890255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3074.282703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3078.402531] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3084.773406] LustreError: 91726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3084.794032] LustreError: 91726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3087.768774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.416872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3094.183996] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3106.191513] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3112.986275] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3122.764064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3123.236541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3132.232395] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3140.695455] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3144.760105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3157.992750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3158.000647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3158.033520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3159.011772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3159.026140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.055894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3160.058162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3160.065220] Lustre: Skipped 1 previous similar message [ 3165.165809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3165.170612] Lustre: Skipped 1 previous similar message [ 3170.278363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3170.291665] Lustre: Skipped 1 previous similar message [ 3172.319196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3172.559615] Lustre: server umount lustre-OST0000 complete [ 3180.523489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3180.540637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3180.561552] Lustre: Skipped 1 previous similar message [ 3180.584889] Lustre: Skipped 1 previous similar message [ 3182.721150] Lustre: server umount lustre-MDT0000 complete [ 3185.633114] LustreError: 91726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3185.660961] LustreError: 91726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3186.523679] LustreError: 91707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829298 with bad export cookie 6716239995742807544 [ 3186.545024] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3186.546742] LustreError: 91707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3186.914692] Lustre: server umount lustre-MDT0001 complete [ 3197.545839] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3200.676996] Key type lgssc unregistered [ 3201.059525] LNet: 95291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3201.064987] LNetError: 95291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3201.081266] LNet: Removed LNI 192.168.201.112@tcp [ 3202.122483] Key type .llcrypt unregistered [ 3202.126402] Key type ._llcrypt unregistered [ 3226.141939] Key type ._llcrypt registered [ 3226.144382] Key type .llcrypt registered [ 3226.237517] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3228.036151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3228.071223] alg: No test for adler32 (adler32-zlib) [ 3229.121500] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 3229.356994] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3231.039318] Key type lgssc registered [ 3232.508530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3240.161545] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3240.187825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3255.839446] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3260.962307] LustreError: 96676:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [ 3287.647935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3295.122374] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3303.372163] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3309.004415] Lustre: server umount lustre-OST0000 complete [ 3319.855653] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 07:17:09 (1787829429) [ 3327.632546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3328.283119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3333.168796] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3333.613594] LustreError: 98315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.632905] LustreError: 98315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3338.733650] LustreError: 98314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3341.973765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.321416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3346.797264] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3354.241823] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3359.476924] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3365.982661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3366.312084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.776657] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3373.510091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3379.923683] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3392.995432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3392.999797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.011268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3394.022184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3394.022913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3394.025921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3396.809534] Lustre: server umount lustre-OST0000 complete [ 3403.744420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3403.755644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.772850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3403.779764] Lustre: Skipped 1 previous similar message [ 3406.573132] Lustre: server umount lustre-MDT0000 complete [ 3409.380308] LustreError: 99555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3409.401570] LustreError: 99555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3411.078256] LustreError: 98295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829523 with bad export cookie 6505412081757051736 [ 3411.087256] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3411.460609] Lustre: server umount lustre-MDT0001 complete [ 3424.316890] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3428.105277] Key type lgssc unregistered [ 3428.644566] LNet: 102009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3428.656561] LNetError: 102009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3428.680370] LNet: Removed LNI 192.168.201.112@tcp [ 3430.078562] Key type .llcrypt unregistered [ 3430.080469] Key type ._llcrypt unregistered [ 3477.296453] Key type ._llcrypt registered [ 3477.300646] Key type .llcrypt registered [ 3477.445935] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3479.451217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3479.591797] alg: No test for adler32 (adler32-zlib) [ 3481.065451] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 3481.572722] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3483.263406] Key type lgssc registered [ 3485.260646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3494.906534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3494.939587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3496.592436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3502.290804] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3506.145812] LustreError: 104198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3506.168368] LustreError: 104198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3511.269279] LustreError: 104197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3512.076307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3512.693882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3518.844687] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3529.324423] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3534.975982] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3542.359285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3542.714248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3549.535719] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3557.698595] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3566.311818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3579.872812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3579.894568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3579.908532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3581.922958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3581.926299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3581.933220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3581.969812] Lustre: Skipped 1 previous similar message [ 3584.081248] Lustre: server umount lustre-OST0000 complete [ 3589.600566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3589.610121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3589.631042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3592.164836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3592.167429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.118781] Lustre: server umount lustre-MDT0000 complete [ 3597.286070] LustreError: 104927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.308821] LustreError: 104927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3597.947215] LustreError: 104179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829709 with bad export cookie 6416426522508151206 [ 3597.953963] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3597.958994] LustreError: 104179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3598.408918] Lustre: server umount lustre-MDT0001 complete [ 3609.011698] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3611.703549] Key type lgssc unregistered [ 3612.056034] LNet: 107772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3612.063546] LNetError: 107772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3612.082417] LNet: Removed LNI 192.168.201.112@tcp [ 3613.058250] Key type .llcrypt unregistered [ 3613.065072] Key type ._llcrypt unregistered [ 3631.620668] Key type ._llcrypt registered [ 3631.623702] Key type .llcrypt registered [ 3631.733241] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3632.890584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3633.172393] alg: No test for adler32 (adler32-zlib) [ 3634.478145] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 3634.706776] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3636.391443] Key type lgssc registered [ 3637.468511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3648.433541] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3648.461975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3664.100157] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3669.215556] LustreError: 109014:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [ 3695.903577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3702.203391] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3709.866281] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3716.986424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3717.235334] LustreError: MGC192.168.201.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3717.246437] LustreError: 110038:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3717.254213] LustreError: 110038:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3717.264601] LustreError: 110038:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3717.326337] Lustre: server umount lustre-MDT0000 complete [ 3717.329860] LustreError: 110038:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3735.150663] Lustre: server umount lustre-OST0000 complete [ 3745.354857] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 3755.162370] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3789.361266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3789.663779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3789.700752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3789.806791] Lustre: lustre-MDT0000: new disk, initializing [ 3789.861253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3789.877830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3794.347806] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3805.420115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3805.548674] Lustre: 112831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3805.598586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3805.602357] Lustre: Skipped 1 previous similar message [ 3805.651890] Lustre: lustre-MDT0001: new disk, initializing [ 3805.720726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3805.760448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3805.773323] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3810.970875] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3820.352739] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3826.095337] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3832.729666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3833.058862] Lustre: lustre-OST0000: new disk, initializing [ 3833.063598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3833.072987] Lustre: 114797:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3833.164778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3838.673387] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3843.134175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3843.150438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3843.288390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3847.477914] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3852.697148] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3852.908212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3858.144569] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3858.345489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3864.047094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3864.054252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3864.072963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3864.088078] Lustre: Skipped 1 previous similar message [ 3869.167243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3869.174682] Lustre: Skipped 1 previous similar message [ 3874.273945] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3874.275791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3874.313496] Lustre: Skipped 1 previous similar message [ 3874.576521] Lustre: server umount lustre-OST0000 complete [ 3882.472163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3882.480452] LustreError: Skipped 1 previous similar message [ 3882.482452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3882.490117] Lustre: Skipped 1 previous similar message [ 3882.492170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3884.522061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.447614] Lustre: server umount lustre-MDT0000 complete [ 3891.459327] LustreError: 112821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830003 with bad export cookie 18374496517787865905 [ 3891.474624] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3891.966432] Lustre: server umount lustre-MDT0001 complete [ 3902.749873] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 07:26:53 (1787830013) [ 3906.086626] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3924.758397] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 3934.313125] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3970.335514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.548200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3970.571762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3970.633471] Lustre: lustre-MDT0000: new disk, initializing [ 3970.688083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3970.707293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3975.472761] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3987.102203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3987.285576] Lustre: 119702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3987.345596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3987.352579] Lustre: Skipped 1 previous similar message [ 3987.465546] Lustre: lustre-MDT0001: new disk, initializing [ 3987.595831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3987.664310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3987.671132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3992.691311] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4002.093663] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4008.161836] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4015.368874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4015.592673] Lustre: lustre-OST0000: new disk, initializing [ 4015.596862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4015.601212] Lustre: 121667:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4015.671789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.463527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4022.475911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4022.662225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4022.870947] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4033.714143] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4039.022362] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4039.389556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4045.111445] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4045.477316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4048.360221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4048.379610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4048.417751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4048.433881] Lustre: Skipped 3 previous similar messages [ 4058.598827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4058.604652] Lustre: Skipped 3 previous similar messages [ 4062.175627] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4062.468705] Lustre: server umount lustre-OST0000 complete [ 4068.832991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4068.868549] Lustre: Skipped 1 previous similar message [ 4072.923569] Lustre: server umount lustre-MDT0000 complete [ 4073.961642] LustreError: 119712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.974476] LustreError: 119712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4077.051991] LustreError: 121677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830188 with bad export cookie 18374496517787866787 [ 4077.054432] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4077.058091] LustreError: 121677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4077.497094] Lustre: server umount lustre-MDT0001 complete [ 4088.671027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4089.249557] LustreError: 123884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4089.341282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4093.588452] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4094.434977] LustreError: 123885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4099.571687] LustreError: 123884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4101.927310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.640469] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4114.919237] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4120.788656] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4128.370543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4135.051654] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4142.563737] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4152.353816] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4158.439669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4158.454978] Lustre: Skipped 1 previous similar message [ 4158.456210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4158.465988] LustreError: Skipped 1 previous similar message [ 4158.470979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4158.475703] Lustre: Skipped 2 previous similar messages [ 4162.990995] Lustre: server umount lustre-OST0000 complete [ 4173.089671] Lustre: server umount lustre-MDT0000 complete [ 4173.793718] LustreError: 126002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4173.822104] LustreError: 126002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4176.407035] LustreError: 126476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830288 with bad export cookie 18374496517787867669 [ 4176.411923] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4176.421503] LustreError: 126476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4176.704615] Lustre: server umount lustre-MDT0001 complete [ 4188.089796] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 4191.171695] Key type lgssc unregistered [ 4191.501885] LNet: 127520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4191.504891] LNetError: 127520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4191.526920] LNet: Removed LNI 192.168.201.112@tcp [ 4192.632275] Key type .llcrypt unregistered [ 4192.633980] Key type ._llcrypt unregistered [ 4211.500951] Key type ._llcrypt registered [ 4211.503637] Key type .llcrypt registered [ 4211.611298] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 4227.076544] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4227.946764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4227.974240] alg: No test for adler32 (adler32-zlib) [ 4229.123859] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 4229.412229] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 4231.103190] Key type lgssc registered [ 4232.478377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4270.422801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4270.455705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4271.866162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4271.928405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4272.084840] Lustre: lustre-MDT0000: new disk, initializing [ 4272.174448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4272.194356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4276.701790] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4286.583518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4286.691397] Lustre: 130683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4286.750075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4286.751771] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4286.754375] Lustre: Skipped 1 previous similar message [ 4286.836418] Lustre: lustre-MDT0001: new disk, initializing [ 4286.950040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4287.004192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4287.012615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4291.998795] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4301.343501] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4307.181695] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4315.796052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4316.065401] Lustre: lustre-OST0000: new disk, initializing [ 4316.074078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4316.079755] Lustre: 132649:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4316.234510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4323.530177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4323.550825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4323.706864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4324.021893] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4335.424334] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4341.330627] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4341.542731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4347.401642] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4347.734249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4354.528359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4354.543303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4354.565938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4354.577623] Lustre: Skipped 1 previous similar message [ 4359.676220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4359.689024] Lustre: Skipped 1 previous similar message [ 4364.255307] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4364.641763] Lustre: server umount lustre-OST0000 complete [ 4369.888486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4369.913721] Lustre: Skipped 1 previous similar message [ 4369.937989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4369.947208] Lustre: Skipped 1 previous similar message [ 4375.016269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4375.039165] Lustre: Skipped 1 previous similar message [ 4375.871890] Lustre: server umount lustre-MDT0000 complete [ 4380.843032] LustreError: 130676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830492 with bad export cookie 4212274596143653585 [ 4380.861515] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4380.873781] LustreError: 130676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4381.324565] Lustre: server umount lustre-MDT0001 complete [ 4390.542655] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 07:35:00 (1787830500) [ 4397.587133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4397.919961] LustreError: 135062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.935374] LustreError: 135062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4397.984359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4403.055897] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4403.175852] LustreError: 135063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.277117] LustreError: 135062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.386514] LustreError: 135063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.740880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4413.185980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4417.973529] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4425.916915] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4433.105726] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4438.304729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4438.307319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4438.329812] Lustre: Skipped 2 previous similar messages [ 4443.559351] Lustre: server umount lustre-MDT0000 complete [ 4443.618046] LustreError: 136263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4443.646952] LustreError: 136263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4447.209984] LustreError: 135044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830559 with bad export cookie 4212274596143654474 [ 4447.220462] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4447.475308] Lustre: server umount lustre-MDT0001 complete [ 4456.576750] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 07:36:06 (1787830566) [ 4464.404367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4480.159753] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4485.279490] LustreError: 137480:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [ 4511.007329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4518.130365] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4526.804075] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4529.298626] Lustre: server umount lustre-OST0000 complete [ 4538.183534] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 07:37:28 (1787830648) [ 4546.833163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4547.604166] LustreError: 138926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4547.721295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4552.606393] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4561.899091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.188540] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4574.312377] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4580.051270] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4586.801581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4587.275142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4587.281213] Lustre: Skipped 1 previous similar message [ 4594.995693] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4602.596836] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4613.089719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4613.095505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4613.110614] Lustre: Skipped 1 previous similar message [ 4613.137298] Lustre: Skipped 3 previous similar messages [ 4616.509769] Lustre: server umount lustre-MDT0000 complete [ 4618.212807] LustreError: 138926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4618.233369] LustreError: 138926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4620.765167] LustreError: 138908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830732 with bad export cookie 4212274596143655139 [ 4620.773148] LustreError: 138908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4620.779500] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4621.026883] Lustre: server umount lustre-MDT0001 complete [ 4635.172469] Lustre: server umount lustre-OST0000 complete [ 4645.387479] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 07:39:15 (1787830755) [ 4653.587102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4654.131404] LustreError: 142578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4654.230486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4658.934291] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4667.211836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4672.067089] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4679.281558] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4684.717588] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4691.100484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4697.905393] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4699.713913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4705.656347] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4709.804379] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4710.012678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4714.110559] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4714.313362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4718.056916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4718.066774] LustreError: Skipped 1 previous similar message [ 4718.069538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4718.097356] Lustre: Skipped 1 previous similar message [ 4718.103988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4722.894442] Lustre: server umount lustre-OST0000 complete [ 4728.800744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4728.814302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4728.841407] Lustre: Skipped 1 previous similar message [ 4734.300765] Lustre: server umount lustre-MDT0000 complete [ 4735.457536] LustreError: 142578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4735.488600] LustreError: 142578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4742.699101] LustreError: 142559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830854 with bad export cookie 4212274596143656210 [ 4742.705985] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4742.709584] LustreError: 142559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4743.172419] Lustre: server umount lustre-MDT0001 complete [ 4752.963191] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 07:41:03 (1787830863) [ 4760.908819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4776.671829] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4781.796318] LustreError: 146738:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [ 4807.521286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4807.529218] Lustre: Skipped 2 previous similar messages [ 4815.061665] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4822.686635] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4830.506086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4831.598960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4836.316206] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4844.979524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4850.575346] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4858.873828] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4864.874821] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4869.785455] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4869.987773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4874.696158] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4875.055180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4881.381614] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.381965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4881.405118] Lustre: Skipped 1 previous similar message [ 4881.410200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4881.440815] Lustre: Skipped 7 previous similar messages [ 4883.552173] Lustre: server umount lustre-MDT0000 complete [ 4886.516816] LustreError: 146763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4886.541473] LustreError: 146763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 4887.338821] LustreError: 146745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830999 with bad export cookie 4212274596143657232 [ 4887.353244] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4887.609948] Lustre: server umount lustre-MDT0001 complete [ 4901.902592] Lustre: server umount lustre-OST0000 complete [ 4909.704556] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 07:43:40 (1787831020) [ 4916.131062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4931.680820] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4936.799835] LustreError: 150933:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [ 4969.242790] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4976.774779] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4984.823118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4985.538690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4990.439605] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5000.047748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.582930] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5013.776486] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5018.937482] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5027.309845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5027.529631] Lustre: lustre-OST0001: new disk, initializing [ 5027.532955] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5027.537687] Lustre: 153808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5034.112881] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5036.643539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5036.660498] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5036.837109] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5043.079103] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5047.633296] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5047.948110] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5052.928045] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5053.188860] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5057.001115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5057.014955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5057.017059] LustreError: Skipped 1 previous similar message [ 5057.018936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5057.043477] Lustre: Skipped 3 previous similar messages [ 5061.437275] Lustre: server umount lustre-OST0000 complete [ 5067.756739] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5071.530356] Lustre: server umount lustre-OST0001 complete [ 5081.282947] Lustre: server umount lustre-MDT0000 complete [ 5085.268728] LustreError: 150940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831197 with bad export cookie 4212274596143658184 [ 5085.282475] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5085.284602] LustreError: 150940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5085.992852] Lustre: server umount lustre-MDT0001 complete [ 5114.810197] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 07:47:05 (1787831225) [ 5116.703856] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5117.913128] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5120.226295] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 07:47:10 (1787831230) [ 5128.028985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5128.382934] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5128.418884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5128.590037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5128.594796] Lustre: Skipped 6 previous similar messages [ 5132.589593] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5141.477855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5141.537801] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5141.608101] Lustre: 157370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5146.031302] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5152.960772] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5158.185026] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5164.590714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5164.717742] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5171.264399] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5172.875622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5178.543921] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5182.937520] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5183.119502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5187.940661] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5188.240773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5193.188967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5193.194674] LustreError: Skipped 2 previous similar messages [ 5193.198635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5193.209600] Lustre: Skipped 4 previous similar messages [ 5196.710323] Lustre: server umount lustre-OST0000 complete [ 5220.360138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5227.429303] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5233.968056] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.985619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.000815] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5236.055785] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5241.687687] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5242.008220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5248.101886] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5248.332300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5254.097804] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff90c703322000.ost_server_uuid 50 [ 5255.935538] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff90c703322000.ost_server_uuid in FULL state after 0 sec [ 5256.151547] LustreError: 161485:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.12@tcp) returned error from blocking AST (req@000000002496b7bf x1874675987907712 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000c8868d8e/0x3a75016d778b7e22 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.12@tcp remote: 0xca5d7b9110bf495 expref: 5 pid: 157375 timeout: 5356 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5256.197921] LustreError: lustre-MDT0000: A client on nid 192.168.201.12@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5256.206293] LustreError: 157365:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.12@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000c8868d8e/0x3a75016d778b7e22 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.12@tcp remote: 0xca5d7b9110bf495 expref: 6 pid: 157375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5261.798837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5261.801802] LustreError: Skipped 1 previous similar message [ 5266.734348] Lustre: server umount lustre-OST0000 complete [ 5276.867715] Lustre: server umount lustre-MDT0000 complete [ 5280.224450] LustreError: 160173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5280.251152] LustreError: 160173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 5282.140521] LustreError: 157362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831394 with bad export cookie 4212274596143659367 [ 5282.155193] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5282.164463] LustreError: 157362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5288.732076] Lustre: server umount lustre-MDT0001 complete [ 5301.336682] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5304.519956] Key type lgssc unregistered [ 5304.892738] LNet: 163489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5304.899623] LNetError: 163489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5305.961779] LNet: Removed LNI 192.168.201.112@tcp [ 5307.061453] Key type .llcrypt unregistered [ 5307.063125] Key type ._llcrypt unregistered [ 5330.867425] Key type ._llcrypt registered [ 5330.873830] Key type .llcrypt registered [ 5331.020676] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5332.176310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5332.191631] alg: No test for adler32 (adler32-zlib) [ 5333.498972] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 5333.824290] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5335.551225] Key type lgssc registered [ 5336.995244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5345.486016] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5345.520661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5346.895346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5351.860660] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5357.026052] LustreError: 164899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5357.061515] LustreError: 164899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5361.006815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.379407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5365.412260] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5372.043545] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5377.995414] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5385.652675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5386.144866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5391.291137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5393.252680] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5401.051083] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5408.868760] Lustre: Failing over lustre-MDT0000 [ 5409.013597] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 5409.175967] Lustre: server umount lustre-MDT0000 complete [ 5411.809749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5411.813348] LustreError: 164895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5411.829694] Lustre: Skipped 2 previous similar messages [ 5411.860343] LustreError: 164895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5416.929936] LustreError: 164894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5416.943173] LustreError: 164894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5421.118745] LustreError: 164879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.12@tcp arrived at 1787831533 with bad export cookie 15778991027929663142 [ 5422.089910] LustreError: 166741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5422.106027] LustreError: 166741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5427.170884] LustreError: 164895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5427.197300] LustreError: 164895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5428.164478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5428.192368] Lustre: 164213:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831523/real 1787831523] req@0000000028252573 x1874677145299840/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787831539 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5428.224974] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5438.433569] LustreError: 164210:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000930c04be x1874677145305344/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5438.462432] LustreError: 164895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.476940] LustreError: 164895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5438.787752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5438.851909] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5443.375114] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5444.070153] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5444.085358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5445.838229] Lustre: lustre-MDT0000: Denying connection for new client b3669f92-1d7f-4d91-aeb6-c62124cea013 (at 192.168.201.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5454.319065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.327398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5454.354703] Lustre: Skipped 1 previous similar message [ 5459.440205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5459.447367] Lustre: Skipped 2 previous similar messages [ 5460.253792] Lustre: server umount lustre-OST0000 complete [ 5464.202511] LustreError: 168632:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5464.213090] Lustre: 167980:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5464.220714] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5464.235975] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5464.245983] LustreError: 167980:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ffe99889 x1874677145319296/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5464.256947] LustreError: 167980:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5464.262715] LustreError: 167980:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5464.277606] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5464.548435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5464.560682] LustreError: 168344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5464.865860] Lustre: server umount lustre-MDT0000 complete [ 5469.419798] LustreError: 164210:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5469.422393] LustreError: 164881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831581 with bad export cookie 15778991027929663240 [ 5469.428178] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5469.487173] LustreError: 164881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5469.971846] Lustre: server umount lustre-MDT0001 complete [ 5483.831953] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5486.302521] Key type lgssc unregistered [ 5486.534321] LNet: 169390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5486.555446] LNetError: 169390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5486.586726] LNet: Removed LNI 192.168.201.112@tcp [ 5487.272147] Key type .llcrypt unregistered [ 5487.274716] Key type ._llcrypt unregistered [ 5509.576154] Key type ._llcrypt registered [ 5509.581399] Key type .llcrypt registered [ 5509.696971] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5510.626042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5510.741978] alg: No test for adler32 (adler32-zlib) [ 5511.759839] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 5511.987579] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5513.663938] Key type lgssc registered [ 5514.965772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5523.471789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5523.488923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.056994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5530.314142] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5534.180556] LustreError: 170802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5534.194333] LustreError: 170802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5539.307590] LustreError: 170803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5540.532952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5540.852783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5544.654503] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5551.962821] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5558.304312] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5565.265550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5565.624575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5570.750532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5572.142822] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5579.385884] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5586.406757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.407332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5586.446506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5590.913824] Lustre: server umount lustre-OST0000 complete [ 5596.641420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5596.648878] Lustre: Skipped 1 previous similar message [ 5596.653103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5596.655824] Lustre: Skipped 1 previous similar message [ 5601.454540] Lustre: server umount lustre-MDT0000 complete [ 5601.768565] LustreError: 170803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5601.779259] LustreError: 170803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5605.178611] LustreError: 170782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831717 with bad export cookie 16955504874953636833 [ 5605.199593] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5605.202186] LustreError: 170782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5605.639574] Lustre: server umount lustre-MDT0001 complete [ 5616.050558] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5618.371926] Key type lgssc unregistered [ 5618.651908] LNet: 174337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5618.660820] LNetError: 174337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5618.673245] LNet: Removed LNI 192.168.201.112@tcp [ 5619.294385] Key type .llcrypt unregistered [ 5619.297415] Key type ._llcrypt unregistered [ 5650.992698] Key type ._llcrypt registered [ 5650.998446] Key type .llcrypt registered [ 5651.238438] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5653.045081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5653.155399] alg: No test for adler32 (adler32-zlib) [ 5654.799092] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 5655.090746] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5656.769362] Key type lgssc registered [ 5658.436752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5668.077691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5668.105070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5669.794062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.170677] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5679.087535] LustreError: 176041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5679.131328] LustreError: 176041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5684.195383] LustreError: 176042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5684.520045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5684.856591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5689.166655] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5696.736932] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5704.321111] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5712.100443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5712.446272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5714.480875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5718.725314] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5726.837964] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5740.000211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5740.003751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5740.010746] LustreError: Skipped 1 previous similar message [ 5740.017349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5740.024874] Lustre: Skipped 1 previous similar message [ 5740.037863] Lustre: Skipped 1 previous similar message [ 5744.097367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5745.784309] Lustre: server umount lustre-OST0000 complete [ 5751.264890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5751.276264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5751.297159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5751.302415] Lustre: Skipped 1 previous similar message [ 5754.342319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5754.343628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5756.183457] Lustre: server umount lustre-MDT0000 complete [ 5759.469962] LustreError: 176042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.493255] LustreError: 176042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5760.111040] LustreError: 176020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831872 with bad export cookie 8991639937016955067 [ 5760.116510] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5760.119864] LustreError: 176020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5760.423605] Lustre: server umount lustre-MDT0001 complete [ 5770.864622] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5773.568605] Key type lgssc unregistered [ 5773.928710] LNet: 179589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5773.934794] LNetError: 179589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5773.961586] LNet: Removed LNI 192.168.201.112@tcp [ 5774.680165] Key type .llcrypt unregistered [ 5774.682469] Key type ._llcrypt unregistered [ 5797.332464] Key type ._llcrypt registered [ 5797.338375] Key type .llcrypt registered [ 5797.510991] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5799.402036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5799.437670] alg: No test for adler32 (adler32-zlib) [ 5800.710996] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 5800.982680] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5802.695248] Key type lgssc registered [ 5803.905577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5815.958677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5815.983543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5817.407278] Lustre: *** cfs_fail_loc=135, val=0*** [ 5817.494097] LustreError: 181113:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5817.500763] LustreError: 181113:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.112@tcp: cfg command failed: rc = -2 [ 5817.507882] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5817.507882] [ 5817.515193] LustreError: MGC192.168.201.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5817.525292] LustreError: 181074:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5817.531912] LustreError: 181074:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5817.543926] LustreError: 181074:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5817.551855] LustreError: 181074:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5817.651395] Lustre: server umount lustre-MDT0000 complete [ 5817.660335] LustreError: 181074:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5824.262726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5824.693450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5829.350251] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5834.212609] LustreError: 181486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.242825] LustreError: 181486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5837.535421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5837.814810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5842.548762] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5850.710267] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5856.747168] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5859.303463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.313263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5859.320488] Lustre: Skipped 1 previous similar message [ 5862.434577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5862.445994] Lustre: Skipped 1 previous similar message [ 5865.502758] Lustre: server umount lustre-MDT0000 complete [ 5867.491758] LustreError: 181487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5867.515300] LustreError: 181487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5869.117912] LustreError: 181467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831981 with bad export cookie 5308420185040754229 [ 5869.131281] LustreError: 181467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5869.131764] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5869.479345] Lustre: server umount lustre-MDT0001 complete [ 5880.879577] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5883.847517] Key type lgssc unregistered [ 5884.239557] LNet: 183809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5884.245374] LNetError: 183809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5884.257862] LNet: Removed LNI 192.168.201.112@tcp [ 5885.250823] Key type .llcrypt unregistered [ 5885.256333] Key type ._llcrypt unregistered [ 5910.983435] Key type ._llcrypt registered [ 5910.988415] Key type .llcrypt registered [ 5911.121664] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5911.701320] Key type .llcrypt unregistered [ 5911.703227] Key type ._llcrypt unregistered [ 5929.965298] Key type ._llcrypt registered [ 5929.966802] Key type .llcrypt registered [ 5930.106709] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5931.349346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5931.525616] alg: No test for adler32 (adler32-zlib) [ 5932.528950] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 5932.695753] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5934.375281] Key type lgssc registered [ 5935.744978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5943.266044] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5943.290355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5958.816193] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5963.935314] LustreError: 185946:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [ 5990.687225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5996.572148] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6005.893625] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6013.547828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6014.146453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6014.240027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6019.553185] LustreError: 185971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6019.600689] LustreError: 185971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6019.741662] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6024.689482] LustreError: 185972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6025.712816] LustreError: 185971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6025.750155] LustreError: 185971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6030.602072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6030.818666] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6031.047750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6037.312150] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6044.837889] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6051.780564] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6055.091789] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6070.754737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6070.762549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.793915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6070.798943] Lustre: Skipped 1 previous similar message [ 6071.782297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6071.794981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6071.820022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6073.691470] Lustre: server umount lustre-OST0000 complete [ 6082.016964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6082.027687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.039567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6082.044474] Lustre: Skipped 2 previous similar messages [ 6087.655217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.665960] Lustre: Skipped 2 previous similar messages [ 6093.279213] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6093.497781] Lustre: server umount lustre-MDT0000 complete [ 6097.150136] LustreError: 185952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832209 with bad export cookie 7727215114643800696 [ 6097.152582] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6097.160389] LustreError: 185952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6097.694798] Lustre: server umount lustre-MDT0001 complete [ 6109.940684] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 6112.325760] Key type lgssc unregistered [ 6112.572154] LNet: 189927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6112.579365] LNetError: 189927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6113.638662] LNet: Removed LNI 192.168.201.112@tcp [ 6114.493993] Key type .llcrypt unregistered [ 6114.497774] Key type ._llcrypt unregistered [ 6145.505112] Key type ._llcrypt registered [ 6145.507371] Key type .llcrypt registered [ 6145.668062] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6146.848903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6147.158756] alg: No test for adler32 (adler32-zlib) [ 6148.483333] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 6148.889959] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 6150.759423] Key type lgssc registered [ 6152.774981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6162.210976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6162.246344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6163.747178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6167.961524] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6173.157598] LustreError: 191347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6173.180629] LustreError: 191347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6176.806429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.113486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6180.904888] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6187.696385] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6193.755308] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6200.243066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6200.604979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6207.014573] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6208.842773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6215.630790] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6220.809938] Lustre: Failing over lustre-MDT0000 [ 6221.042646] Lustre: server umount lustre-MDT0000 complete [ 6222.564683] LustreError: 191342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6223.329538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6223.340876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6224.352522] LustreError: 192075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6224.356506] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6224.372787] LustreError: 192075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6224.423030] Lustre: Skipped 1 previous similar message [ 6227.660066] LustreError: 191342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6227.695522] LustreError: 191342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6232.759088] LustreError: 191344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6232.773562] LustreError: 191344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6240.735158] Lustre: 190658:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832336/real 1787832336] req@000000001ba1ee89 x1874677999889280/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787832352 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6240.772964] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6240.797352] LustreError: 191344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6240.830618] LustreError: 191344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6242.148356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6250.984899] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb374d9224756a259 [ 6251.009489] Lustre: MGC192.168.201.112@tcp: Connection restored to 0@lo (at 0@lo) [ 6251.285905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6251.345844] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6254.264432] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6256.374805] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6256.648018] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6256.676364] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6256.749287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6261.281836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6269.470929] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6278.624507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6278.631853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6278.665448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6280.716682] Lustre: server umount lustre-OST0000 complete [ 6287.336097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6287.344677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6287.353303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6292.448876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6292.451645] Lustre: Skipped 1 previous similar message [ 6294.712868] Lustre: server umount lustre-MDT0000 complete [ 6297.570206] LustreError: 191348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6297.579322] LustreError: 191348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6299.279781] LustreError: 191329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832411 with bad export cookie 12931199171336512089 [ 6299.285572] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6299.288124] LustreError: 191329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6299.583921] Lustre: server umount lustre-MDT0001 complete [ 6311.364895] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 6314.273831] Key type lgssc unregistered [ 6314.667413] LNet: 196240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6314.678109] LNetError: 196240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6314.690107] LNet: Removed LNI 192.168.201.112@tcp [ 6315.594713] Key type .llcrypt unregistered [ 6315.598873] Key type ._llcrypt unregistered [ 6339.981889] Key type ._llcrypt registered [ 6339.984669] Key type .llcrypt registered [ 6340.071329] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6341.380315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6341.404267] alg: No test for adler32 (adler32-zlib) [ 6342.525059] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 6342.742016] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 6344.391213] Key type lgssc registered [ 6345.384431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6351.748303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6351.790187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6353.214072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6358.645514] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6363.617602] LustreError: 197651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6363.638706] LustreError: 197651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6367.895280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6368.601620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6374.251265] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6383.488819] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6391.232747] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6400.579895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6400.922946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6410.042520] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6419.617742] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6423.501410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6427.555979] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6430.233506] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6452.118255] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6464.484784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6464.492215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6464.497450] LustreError: Skipped 1 previous similar message [ 6464.512326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6464.532114] Lustre: Skipped 1 previous similar message [ 6464.570749] Lustre: Skipped 1 previous similar message [ 6468.930635] Lustre: server umount lustre-OST0000 complete [ 6474.728447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6474.744809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6479.566526] Lustre: server umount lustre-MDT0000 complete [ 6479.844917] LustreError: 197651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6479.868656] LustreError: 197651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6484.222932] LustreError: 197632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832596 with bad export cookie 18296108170107710391 [ 6484.237856] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6484.698886] Lustre: server umount lustre-MDT0001 complete [ 6498.145884] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 6501.449153] Key type lgssc unregistered [ 6501.878885] LNet: 201531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6501.900814] LNetError: 201531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6502.953241] LNet: Removed LNI 192.168.201.112@tcp [ 6504.199143] Key type .llcrypt unregistered [ 6504.202128] Key type ._llcrypt unregistered [ 6528.866761] Key type ._llcrypt registered [ 6528.868701] Key type .llcrypt registered [ 6528.986615] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6530.901251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6530.947978] alg: No test for adler32 (adler32-zlib) [ 6532.102382] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 6532.303944] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 6533.951180] Key type lgssc registered [ 6535.705172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6543.573788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6543.611568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6545.124227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6549.173142] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6555.621264] LustreError: 202798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6555.656134] LustreError: 202798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6558.933085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6559.512786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6564.097436] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6572.257821] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6578.325701] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6587.951046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6588.449738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6590.539074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6596.420284] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6606.535817] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6613.337428] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6626.371385] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6638.407132] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6644.457117] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6655.971383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6655.975478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6655.985374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6661.092974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6661.096862] Lustre: Skipped 2 previous similar messages [ 6661.708478] Lustre: server umount lustre-OST0000 complete [ 6666.208848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6666.223945] Lustre: Skipped 1 previous similar message [ 6666.229734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6666.236835] Lustre: Skipped 1 previous similar message [ 6671.403231] LustreError: 203526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6671.491278] Lustre: server umount lustre-MDT0000 complete [ 6675.248141] LustreError: 204644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832787 with bad export cookie 12771018877618356964 [ 6675.249908] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6675.257018] LustreError: 204644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6675.596896] Lustre: server umount lustre-MDT0001 complete [ 6687.038206] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 6689.640639] Key type lgssc unregistered [ 6689.893090] LNet: 207775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6689.899286] LNetError: 207775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6689.918540] LNet: Removed LNI 192.168.201.112@tcp [ 6690.875180] Key type .llcrypt unregistered [ 6690.877229] Key type ._llcrypt unregistered [ 6714.312326] Key type ._llcrypt registered [ 6714.313870] Key type .llcrypt registered [ 6714.455252] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6715.515153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6715.664779] alg: No test for adler32 (adler32-zlib) [ 6716.799196] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 6717.005105] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 6718.655160] Key type lgssc registered [ 6719.831869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6727.550814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6727.580750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.115939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6733.215183] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6739.425671] LustreError: 209188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6739.444576] LustreError: 209188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6743.266322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.819843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6751.240421] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6762.860874] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6768.821979] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6776.356486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6776.666704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.156531] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6790.199230] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6799.212143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6814.706119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6814.725219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.743860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6814.751423] Lustre: Skipped 1 previous similar message [ 6818.972573] Lustre: server umount lustre-OST0000 complete [ 6824.928864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6824.938657] Lustre: Skipped 1 previous similar message [ 6824.948627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6824.954397] Lustre: Skipped 1 previous similar message [ 6829.767803] Lustre: server umount lustre-MDT0000 complete [ 6830.050405] LustreError: 209189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6830.080323] LustreError: 209189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6834.051918] LustreError: 209169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832945 with bad export cookie 14030554030661846273 [ 6834.053119] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6834.060938] LustreError: 209169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6834.347731] Lustre: server umount lustre-MDT0001 complete [ 6845.124631] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 6848.222160] Key type lgssc unregistered [ 6848.551653] LNet: 212898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6848.576408] LNetError: 212898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6848.607146] LNet: Removed LNI 192.168.201.112@tcp [ 6849.631676] Key type .llcrypt unregistered [ 6849.634285] Key type ._llcrypt unregistered [ 6871.605708] Key type ._llcrypt registered [ 6871.608954] Key type .llcrypt registered [ 6871.726805] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6873.633364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6873.685127] alg: No test for adler32 (adler32-zlib) [ 6875.338983] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 6875.741439] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 6877.585710] Key type lgssc registered [ 6880.888395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6893.425627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6893.436547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.892647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6899.662867] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6904.300543] LustreError: 214168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6904.321194] LustreError: 214168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6908.195305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6908.643406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6913.648341] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6921.286372] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6927.918311] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6934.652240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6935.185966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6939.320626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6942.529689] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6951.881968] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6969.316754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6969.325900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6969.348330] Lustre: Skipped 1 previous similar message [ 6969.368372] Lustre: Skipped 1 previous similar message [ 6972.667110] Lustre: server umount lustre-OST0000 complete [ 6979.557506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6979.561862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6979.569986] Lustre: Skipped 1 previous similar message [ 6979.590983] Lustre: Skipped 1 previous similar message [ 6982.939203] Lustre: server umount lustre-MDT0000 complete [ 6984.674850] LustreError: 214167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6984.694703] LustreError: 214167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6987.147599] LustreError: 214148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833099 with bad export cookie 15502081127083388202 [ 6987.150076] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6987.162232] LustreError: 214148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6987.708356] Lustre: server umount lustre-MDT0001 complete [ 7000.757155] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 7004.727943] Key type lgssc unregistered [ 7005.243255] LNet: 217853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7005.259370] LNetError: 217853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7005.288867] LNet: Removed LNI 192.168.201.112@tcp [ 7006.529963] Key type .llcrypt unregistered [ 7006.531741] Key type ._llcrypt unregistered [ 7026.131941] Key type ._llcrypt registered [ 7026.134549] Key type .llcrypt registered [ 7026.255721] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7028.021258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7028.081683] alg: No test for adler32 (adler32-zlib) [ 7029.325200] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 7029.639190] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 7031.319352] Key type lgssc registered [ 7032.677094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7043.621345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7043.651870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7045.223697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7052.175442] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7055.333070] LustreError: 219122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7055.374620] LustreError: 219122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7060.454993] LustreError: 219123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7064.553085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7065.357895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7069.621260] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7076.496834] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7082.858712] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7090.220085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7090.583501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7091.639411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7096.858153] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7104.594059] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7114.902612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7115.078127] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7115.454100] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7122.876852] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7131.239864] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7132.726590] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7144.519822] Lustre: Permanently deactivating lustre-OST0001 [ 7144.535536] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7153.239744] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7161.005073] Lustre: Permanently reactivating lustre-OST0001 [ 7161.026042] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7161.031126] Lustre: Skipped 2 previous similar messages [ 7170.172437] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7170.185533] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7170.200939] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7170.215926] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7170.270159] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7175.649587] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7175.651898] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7175.654412] LustreError: Skipped 1 previous similar message [ 7175.661074] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7175.668905] Lustre: Skipped 2 previous similar messages [ 7175.682096] Lustre: Skipped 1 previous similar message [ 7178.226888] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7178.241230] Lustre: Skipped 1 previous similar message [ 7181.623500] Lustre: server umount lustre-OST0001 complete [ 7183.335702] LustreError: 220947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7183.353280] LustreError: 220947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7188.453601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7188.459766] LustreError: 221029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7188.475746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7188.491065] LustreError: 221029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7188.510868] Lustre: Skipped 1 previous similar message [ 7191.831427] Lustre: server umount lustre-OST0000 complete [ 7198.178573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7198.191674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7198.206021] Lustre: Skipped 1 previous similar message [ 7198.217397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7201.834768] Lustre: server umount lustre-MDT0000 complete [ 7203.808905] LustreError: 219853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7203.822190] LustreError: 219853:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7205.501782] LustreError: 219860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833317 with bad export cookie 3679600801324715311 [ 7205.506747] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7205.514835] LustreError: 219860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7205.861632] Lustre: server umount lustre-MDT0001 complete [ 7216.764185] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 7219.711602] Key type lgssc unregistered [ 7220.039508] LNet: 224223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7220.048408] LNetError: 224223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7220.077819] LNet: Removed LNI 192.168.201.112@tcp [ 7221.282967] Key type .llcrypt unregistered [ 7221.284617] Key type ._llcrypt unregistered [ 7266.420196] Key type ._llcrypt registered [ 7266.424462] Key type .llcrypt registered [ 7266.521793] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7267.608668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7267.675060] alg: No test for adler32 (adler32-zlib) [ 7268.752438] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 7268.998209] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 7270.703173] Key type lgssc registered [ 7271.714442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7278.447464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7278.463648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7279.785259] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7279.833679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7279.990136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7284.932741] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7293.189020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7293.315274] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7293.344978] Lustre: 226428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7293.644455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7298.575244] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7307.359122] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7313.720206] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7320.255795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7320.394603] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7320.660514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7328.455901] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7329.477442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7336.056086] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7339.638849] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7343.449276] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7352.292372] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7362.185169] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7376.455831] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7376.461860] Lustre: Skipped 1 previous similar message [ 7395.989921] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7395.997577] Lustre: Skipped 2 previous similar messages [ 7436.038812] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7436.042389] Lustre: Skipped 4 previous similar messages [ 7500.219077] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7500.223958] Lustre: Skipped 7 previous similar messages [ 7508.962507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7508.978652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7508.983600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7509.008284] Lustre: Skipped 1 previous similar message [ 7509.022848] Lustre: Skipped 1 previous similar message [ 7512.040767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7512.048925] Lustre: Skipped 1 previous similar message [ 7513.650581] Lustre: server umount lustre-OST0000 complete [ 7517.153175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7517.178671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7517.190717] Lustre: Skipped 1 previous similar message [ 7522.274445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7522.285872] Lustre: Skipped 1 previous similar message [ 7522.923425] Lustre: server umount lustre-MDT0000 complete [ 7525.876549] LustreError: 229324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833637 with bad export cookie 5029899596365514480 [ 7525.889221] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7526.275537] Lustre: server umount lustre-MDT0001 complete [ 7535.098631] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 7538.046989] Key type lgssc unregistered [ 7538.407551] LNet: 231214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7538.423918] LNetError: 231214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7539.512349] LNet: Removed LNI 192.168.201.112@tcp [ 7540.303192] Key type .llcrypt unregistered [ 7540.305305] Key type ._llcrypt unregistered [ 7562.594541] Key type ._llcrypt registered [ 7562.600742] Key type .llcrypt registered [ 7562.684287] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7563.884825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7564.066676] alg: No test for adler32 (adler32-zlib) [ 7565.137714] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 7565.423166] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 7567.143953] Key type lgssc registered [ 7568.228271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7575.440510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7575.455246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7576.865862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7582.303397] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7586.278080] LustreError: 232631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7586.307647] LustreError: 232631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7591.398730] LustreError: 232630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7591.659270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7592.162221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7597.424415] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7605.523973] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7612.815945] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7619.477280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7619.996025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7622.093345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7627.107203] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7635.663193] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7656.936328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7656.936563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7656.987048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7662.054730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7662.064126] Lustre: Skipped 2 previous similar messages [ 7662.372161] Lustre: server umount lustre-OST0000 complete [ 7667.169108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7667.170685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7667.195217] Lustre: Skipped 2 previous similar messages [ 7672.291422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7672.322743] Lustre: Skipped 2 previous similar messages [ 7672.447111] Lustre: server umount lustre-MDT0000 complete [ 7676.475060] LustreError: 232611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833788 with bad export cookie 12462162590709357870 [ 7676.478592] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7676.487069] LustreError: 232611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7676.935879] Lustre: server umount lustre-MDT0001 complete [ 7686.469258] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 7689.367955] Key type lgssc unregistered [ 7689.699831] LNet: 236322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7689.707495] LNetError: 236322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7689.737724] LNet: Removed LNI 192.168.201.112@tcp [ 7690.482146] Key type .llcrypt unregistered [ 7690.488020] Key type ._llcrypt unregistered [ 7711.395577] Key type ._llcrypt registered [ 7711.397498] Key type .llcrypt registered [ 7711.507561] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7712.403947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7712.496191] alg: No test for adler32 (adler32-zlib) [ 7713.473959] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 7713.651144] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 7715.279224] Key type lgssc registered [ 7716.358745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7986.109555] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 7989.005995] Key type lgssc unregistered [ 7989.296600] LNet: 238345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7989.301203] LNetError: 238345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7989.312639] LNet: Removed LNI 192.168.201.112@tcp [ 7990.124558] Key type .llcrypt unregistered [ 7990.127336] Key type ._llcrypt unregistered [ 8012.554799] Key type ._llcrypt registered [ 8012.559495] Key type .llcrypt registered [ 8012.691596] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 8031.200994] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 8032.284975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8032.443225] alg: No test for adler32 (adler32-zlib) [ 8033.572940] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 8033.867458] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 8035.584707] Key type lgssc registered [ 8036.689950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8063.839846] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7816 sec ========== 08:36:14 (1787834174) [ 8065.776286] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:36:16 (1787834176) === [ 8069.213938] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:36:19 (1787834179) === [ 8111.406615] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 8114.198644] Key type lgssc unregistered [ 8114.470413] LNet: 243585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8114.478530] LNetError: 243585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8114.504227] LNet: Removed LNI 192.168.201.112@tcp [ 8115.216933] Key type .llcrypt unregistered [ 8115.218587] Key type ._llcrypt unregistered