[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 500508589 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003171] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008668] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010017] pid_max: default: 32768 minimum: 301 [ 0.011170] LSM: Security Framework initializing [ 0.012066] Yama: becoming mindful. [ 0.013049] SELinux: Initializing. [ 0.014099] *** VALIDATE selinux *** [ 0.023207] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028341] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031064] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033054] *** VALIDATE tmpfs *** [ 0.034272] *** VALIDATE proc *** [ 0.035273] *** VALIDATE cgroup *** [ 0.036012] *** VALIDATE cgroup2 *** [ 0.037328] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041023] Spectre V2 : User space: Vulnerable [ 0.042014] Speculative Store Bypass: Vulnerable [ 0.045000] debug: unmapping init [mem 0xffffffffa1e59000-0xffffffffa1e60fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048707] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049034] ... version: 2 [ 0.050015] ... bit width: 48 [ 0.051013] ... generic registers: 4 [ 0.052017] ... value mask: 0000ffffffffffff [ 0.053018] ... max period: 00007fffffffffff [ 0.054019] ... fixed-purpose events: 3 [ 0.055018] ... event mask: 000000070000000f [ 0.056308] rcu: Hierarchical SRCU implementation. [ 0.058600] smp: Bringing up secondary CPUs ... [ 0.059703] x86: Booting SMP configuration: [ 0.060031] .... node #0, CPUs: #1 #2 #3 [ 0.064279] smp: Brought up 1 node, 4 CPUs [ 0.066020] smpboot: Max logical packages: 1 [ 0.067027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.239143] node 0 deferred pages initialised in 170ms [ 0.242365] devtmpfs: initialized [ 0.243246] x86/mm: Memory block size: 128MB [ 0.245000] gcov: version magic: 0x41383552 [ 0.249389] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.250108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.251290] pinctrl core: initialized pinctrl subsystem [ 0.252202] [ 0.252928] ************************************************************* [ 0.253024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.254015] ** ** [ 0.255015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.256017] ** ** [ 0.257017] ** This means that this kernel is built to expose internal ** [ 0.258017] ** IOMMU data structures, which may compromise security on ** [ 0.259018] ** your system. ** [ 0.260019] ** ** [ 0.261016] ** If you see this message and you are not debugging the ** [ 0.262014] ** kernel, report this immediately to your vendor! ** [ 0.263019] ** ** [ 0.264014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.265018] ************************************************************* [ 0.267000] NET: Registered protocol family 16 [ 0.268524] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.271083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.274102] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.277165] cpuidle: using governor menu [ 0.279780] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.282700] PCI: Using configuration type 1 for base access [ 0.285136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.295088] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.297038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.301158] cryptd: max_cpu_qlen set to 1000 [ 0.305299] ACPI: Added _OSI(Module Device) [ 0.307028] ACPI: Added _OSI(Processor Device) [ 0.309022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.311023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.316409] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.323371] ACPI: Interpreter enabled [ 0.325091] ACPI: PM: (supports S0 S3 S4 S5) [ 0.326021] ACPI: Using IOAPIC for interrupt routing [ 0.328154] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.332392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.342605] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.345059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.347028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.351116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.357472] acpiphp: Slot [2] registered [ 0.359131] acpiphp: Slot [5] registered [ 0.361378] acpiphp: Slot [6] registered [ 0.364202] acpiphp: Slot [7] registered [ 0.366181] acpiphp: Slot [8] registered [ 0.368182] acpiphp: Slot [9] registered [ 0.371237] acpiphp: Slot [10] registered [ 0.373171] acpiphp: Slot [3] registered [ 0.374183] acpiphp: Slot [4] registered [ 0.376134] acpiphp: Slot [11] registered [ 0.377115] acpiphp: Slot [12] registered [ 0.379125] acpiphp: Slot [13] registered [ 0.381135] acpiphp: Slot [14] registered [ 0.382145] acpiphp: Slot [15] registered [ 0.384110] acpiphp: Slot [16] registered [ 0.385145] acpiphp: Slot [17] registered [ 0.387110] acpiphp: Slot [18] registered [ 0.388086] acpiphp: Slot [19] registered [ 0.389119] acpiphp: Slot [20] registered [ 0.392068] acpiphp: Slot [21] registered [ 0.393106] acpiphp: Slot [22] registered [ 0.394081] acpiphp: Slot [23] registered [ 0.395106] acpiphp: Slot [24] registered [ 0.397124] acpiphp: Slot [25] registered [ 0.399119] acpiphp: Slot [26] registered [ 0.402170] acpiphp: Slot [27] registered [ 0.404119] acpiphp: Slot [28] registered [ 0.406140] acpiphp: Slot [29] registered [ 0.408128] acpiphp: Slot [30] registered [ 0.409109] acpiphp: Slot [31] registered [ 0.411070] PCI host bridge to bus 0000:00 [ 0.412022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.415029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.418024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.421039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.424035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.427031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.429178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.431165] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.435301] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.445755] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.450529] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.454028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.456019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.460037] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.463367] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.465872] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.469054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.472910] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.479023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.492026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.499023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.505819] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.515019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.529018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.549023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.563315] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.574019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.581016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.600025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.611664] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.621022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.630021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.649022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.661592] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.669020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.677023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.698021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.714615] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.722018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.730016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.750019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.765502] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.779020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.787020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.811023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.826536] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.829393] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.831868] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.835652] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.838237] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.843104] iommu: Default domain type: Passthrough [ 0.845517] SCSI subsystem initialized [ 0.847171] ACPI: bus type USB registered [ 0.849132] usbcore: registered new interface driver usbfs [ 0.851099] usbcore: registered new interface driver hub [ 0.853097] usbcore: registered new device driver usb [ 0.855200] pps_core: LinuxPPS API ver. 1 registered [ 0.857020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.861083] PTP clock support registered [ 0.863159] EDAC MC: Ver: 3.0.0 [ 0.865157] PCI: Using ACPI for IRQ routing [ 0.868029] NetLabel: Initializing [ 0.870014] NetLabel: domain hash size = 128 [ 0.871012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.874096] NetLabel: unlabeled traffic allowed by default [ 0.877106] vgaarb: loaded [ 0.878318] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.880016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.889656] clocksource: Switched to clocksource kvm-clock [ 1.001197] VFS: Disk quotas dquot_6.6.0 [ 1.003220] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.006230] *** VALIDATE ramfs *** [ 1.007677] *** VALIDATE hugetlbfs *** [ 1.010300] pnp: PnP ACPI init [ 1.012952] pnp: PnP ACPI: found 6 devices [ 1.030163] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.034132] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.036594] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.039009] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.041574] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.044411] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.047154] NET: Registered protocol family 2 [ 1.049767] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.055261] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.059360] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.066803] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.071331] TCP: Hash tables configured (established 65536 bind 65536) [ 1.074780] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.078586] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.081818] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.084776] NET: Registered protocol family 1 [ 1.088128] RPC: Registered named UNIX socket transport module. [ 1.090694] RPC: Registered udp transport module. [ 1.092596] RPC: Registered tcp transport module. [ 1.094663] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.097369] NET: Registered protocol family 44 [ 1.099027] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.101450] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.103864] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.106423] PCI: CLS 0 bytes, default 64 [ 1.108375] Unpacking initramfs... [ 2.520316] debug: unmapping init [mem 0xffff9a8d3cc54000-0xffff9a8d3ffbffff] [ 2.525319] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.527821] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.531513] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.039700] Initialise system trusted keyrings [ 3.041667] Key type blacklist registered [ 3.043776] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.054481] zbud: loaded [ 3.059984] *** VALIDATE nfs *** [ 3.061561] *** VALIDATE nfs4 *** [ 3.063603] pstore: using deflate compression [ 3.067591] Platform Keyring initialized [ 3.182618] NET: Registered protocol family 38 [ 3.186238] Key type asymmetric registered [ 3.187748] Asymmetric key parser 'x509' registered [ 3.189642] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.192561] io scheduler mq-deadline registered [ 3.194209] io scheduler kyber registered [ 3.195763] io scheduler bfq registered [ 3.197534] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.200776] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.203791] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.206730] ACPI: Power Button [PWRF] [ 3.212439] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.219195] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.237533] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.246500] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.272196] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.301137] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.329354] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.334796] Non-volatile memory driver v1.3 [ 3.336877] Linux agpgart interface v0.103 [ 3.370378] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.373774] vda: detected capacity change from 0 to 74698752 [ 3.389598] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.393353] vdb: detected capacity change from 0 to 1073741824 [ 3.411868] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.415962] vdc: detected capacity change from 0 to 2621440000 [ 3.433480] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.436823] vdd: detected capacity change from 0 to 2621440000 [ 3.457085] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.460698] vde: detected capacity change from 0 to 4294967296 [ 3.476937] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.481134] vdf: detected capacity change from 0 to 4294967296 [ 3.488468] libphy: Fixed MDIO Bus: probed [ 3.494237] usbcore: registered new interface driver usbserial_generic [ 3.496699] usbserial: USB Serial support registered for generic [ 3.498712] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.505168] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.507265] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.510111] mousedev: PS/2 mouse device common for all mice [ 3.513383] rtc_cmos 00:05: RTC can wake from S4 [ 3.516813] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.521700] rtc_cmos 00:05: registered as rtc0 [ 3.524513] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.525354] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.528048] intel_pstate: CPU model not supported [ 3.534549] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.536498] hid: raw HID events driver (C) Jiri Kosina [ 3.545128] usbcore: registered new interface driver usbhid [ 3.547480] usbhid: USB HID core driver [ 3.549526] drop_monitor: Initializing network drop monitor service [ 3.552631] Initializing XFRM netlink socket [ 3.555095] NET: Registered protocol family 10 [ 3.558552] Segment Routing with IPv6 [ 3.560171] NET: Registered protocol family 17 [ 3.562951] mpls_gso: MPLS GSO support [ 3.569549] RAS: Correctable Errors collector initialized. [ 3.572053] AVX version of gcm_enc/dec engaged. [ 3.574043] AES CTR mode by8 optimization enabled [ 3.653397] sched_clock: Marking stable (3653366996, 0)->(4655801558, -1002434562) [ 3.659861] registered taskstats version 1 [ 3.662468] Loading compiled-in X.509 certificates [ 3.664848] zswap: loaded using pool lzo/zbud [ 3.693634] Key type big_key registered [ 3.707326] Key type encrypted registered [ 3.709042] ima: No TPM chip found, activating TPM-bypass! [ 3.710914] ima: Allocated hash algorithm: sha1 [ 3.712375] ima: No architecture policies found [ 3.714091] evm: Initialising EVM extended attributes: [ 3.715758] evm: security.selinux [ 3.716926] evm: security.ima [ 3.717766] evm: security.capability [ 3.719093] evm: HMAC attrs: 0x1 [ 3.721984] rtc_cmos 00:05: setting system clock to 2026-09-03 13:25:50 UTC (1788441950) [ 3.728115] debug: unmapping init [mem 0xffffffffa2e03000-0xffffffffa2ffffff] [ 3.731239] debug: unmapping init [mem 0xffffffffa1b82000-0xffffffffa1e58fff] [ 3.740126] Write protecting the kernel read-only data: 28672k [ 3.743468] debug: unmapping init [mem 0xffffffffa0203000-0xffffffffa03fffff] [ 3.746173] debug: unmapping init [mem 0xffffffffa0b14000-0xffffffffa0bfffff] [ 3.785458] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.796805] systemd[1]: Detected virtualization kvm. [ 3.798786] systemd[1]: Detected architecture x86-64. [ 3.800979] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.833514] systemd[1]: No hostname configured. [ 3.835617] systemd[1]: Set hostname to . [ 3.838636] random: systemd: uninitialized urandom read (16 bytes read) [ 3.841899] systemd[1]: Initializing machine ID from random generator. [ 3.888795] random: ln: uninitialized urandom read (6 bytes read) [ 3.977196] random: systemd: uninitialized urandom read (16 bytes read) [ 3.980549] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.985950] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.990684] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.661153] device-mapper: uevent: version 1.0.3 [ 4.663950] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK [ 5.392217] random: fast init done ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.481125] virtio_net virtio0 ens2: renamed from eth0 [ 5.525994] scsi host0: ata_piix [ 5.546358] scsi host1: ata_piix [ 5.548410] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.550735] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.015547] dracut-initqueue[592]: RTNETLINK answers: File exists [ 10.227384] random: crng init done [ 10.228916] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.781674] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.973334] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.231703] SELinux: Disabled at runtime. [ 12.299828] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.309783] systemd[1]: Detected virtualization kvm. [ 12.311967] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.821726] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.825485] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.831981] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.836808] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.840809] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.850257] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.858041] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths.[ 12.996755] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.300487] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.629615] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.635806] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.775296] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.788476] EDAC sbridge: Ver: 1.1.2 [ 15.334564] Key type dns_resolver registered [ 15.639718] NFS: Registering the id_resolver key type [ 15.642145] Key type id_resolver registered [ 15.643957] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg413-server login: [ 50.995020] hrtimer: interrupt took 3855861 ns [ 65.393721] libcfs: loading out-of-tree module taints kernel. [ 65.497492] Key type ._llcrypt registered [ 65.501857] Key type .llcrypt registered [ 65.697164] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 85.330961] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 86.934714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 86.945325] alg: No test for adler32 (adler32-zlib) [ 88.189720] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 89.304390] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 91.024244] Key type lgssc registered [ 93.072435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 109.936412] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 155.083730] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 170.158523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 170.203134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.482717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 171.532776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 171.638490] Lustre: lustre-MDT0000: new disk, initializing [ 171.750285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.773372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 176.713468] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.768025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.879140] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 191.909439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 191.913818] Lustre: Skipped 1 previous similar message [ 192.037827] Lustre: lustre-MDT0001: new disk, initializing [ 192.141167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 192.181451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 192.206834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 197.929414] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.133831] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 214.954904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.303244] Lustre: lustre-OST0000: new disk, initializing [ 215.320464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.330850] Lustre: 8456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.424465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.416320] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.840420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 223.854330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 224.008361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 239.956392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.194097] Lustre: lustre-OST0001: new disk, initializing [ 240.198878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 240.224402] Lustre: 9530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.363994] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 247.805731] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.899995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 247.925806] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 247.993626] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 259.514732] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.713381] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 279.661929] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing check_logdir /tmp/testlogs/ [ 285.503905] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing yml_node [ 290.499307] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 293.993929] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 297.111619] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 298.679208] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:30:43 EDT 2026 [ 317.481884] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 319.222329] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 335.339212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.356582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 340.449781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 340.458692] Lustre: Skipped 3 previous similar messages [ 341.953994] Lustre: server umount lustre-MDT0000 complete [ 346.437744] LustreError: 7932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442293 with bad export cookie 9681116937392086500 [ 346.448391] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 346.464828] LustreError: 7932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 350.690500] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.691717] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 350.703841] Lustre: Skipped 3 previous similar messages [ 350.706541] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 350.730587] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 350.753454] Lustre: Skipped 1 previous similar message [ 351.720582] LustreError: 6524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.741040] LustreError: 6524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 352.982521] Lustre: server umount lustre-MDT0001 complete [ 358.025915] Lustre: server umount lustre-OST0000 complete [ 362.753479] Lustre: server umount lustre-OST0001 complete [ 370.307392] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 381.918405] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 424.492896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.816755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 424.839721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 424.972448] Lustre: lustre-MDT0000: new disk, initializing [ 425.035577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.048891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 430.404872] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 442.644438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.777552] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 442.823912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 442.827029] Lustre: Skipped 1 previous similar message [ 442.901714] Lustre: lustre-MDT0001: new disk, initializing [ 443.088884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 443.176281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 443.191017] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 448.466536] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 459.720611] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 466.554305] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 474.499461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 474.916768] Lustre: lustre-OST0000: new disk, initializing [ 474.930623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 474.951324] Lustre: 17052:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 475.076721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.537452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 476.550927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 476.741572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 482.125081] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 492.070966] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 497.452195] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 497.828640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 502.608651] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 502.782301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 507.361580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 507.377302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.393970] Lustre: Skipped 1 previous similar message [ 507.399503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 507.403961] Lustre: Skipped 2 previous similar messages [ 512.483198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.491957] Lustre: Skipped 1 previous similar message [ 519.136217] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 519.473385] Lustre: server umount lustre-OST0000 complete [ 525.285692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 525.292774] LustreError: Skipped 1 previous similar message [ 525.296764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.315515] Lustre: Skipped 1 previous similar message [ 525.320534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 525.331922] Lustre: Skipped 2 previous similar messages [ 530.257709] Lustre: server umount lustre-MDT0000 complete [ 532.964923] LustreError: 15096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.347210] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442482 with bad export cookie 9681116937392088033 [ 535.358712] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.775456] Lustre: server umount lustre-MDT0001 complete [ 540.107374] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:34:44 (1788442484) [ 547.732807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.289875] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.311469] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 548.374848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.285289] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 553.455907] LustreError: 19320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.699257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.420918] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 575.237804] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 581.612861] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 589.002339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 596.176855] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 604.659947] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 616.934458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 616.951568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.984366] Lustre: Skipped 1 previous similar message [ 617.000227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 617.012520] Lustre: Skipped 2 previous similar messages [ 630.240179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 630.500255] Lustre: server umount lustre-OST0000 complete [ 638.435361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.446435] Lustre: Skipped 1 previous similar message [ 641.348029] Lustre: server umount lustre-MDT0000 complete [ 643.566094] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.586559] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 645.347643] LustreError: 19299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442592 with bad export cookie 9681116937392088915 [ 645.361355] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 645.365179] LustreError: 19299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 645.793585] Lustre: server umount lustre-MDT0001 complete [ 658.285059] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 661.367170] Key type lgssc unregistered [ 661.705777] LNet: 22855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.713718] LNetError: 22855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 661.729058] LNet: Removed LNI 192.168.204.113@tcp [ 662.788222] Key type .llcrypt unregistered [ 662.799644] Key type ._llcrypt unregistered [ 687.059960] Key type ._llcrypt registered [ 687.063092] Key type .llcrypt registered [ 687.254169] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 688.155179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 688.173530] alg: No test for adler32 (adler32-zlib) [ 689.251659] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 689.458762] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 691.152328] Key type lgssc registered [ 692.197959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 700.222282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 700.260668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.854333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.063650] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 712.166095] LustreError: 24284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.193246] LustreError: 24284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 716.830570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 717.119390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 721.664798] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 729.736455] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 736.064505] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 743.722373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 744.258982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 746.305725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 751.116061] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 759.762498] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 782.306165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 782.313927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 782.326711] LustreError: Skipped 1 previous similar message [ 782.334751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 782.350879] Lustre: Skipped 1 previous similar message [ 782.375678] Lustre: Skipped 1 previous similar message [ 785.429470] Lustre: server umount lustre-OST0000 complete [ 792.553747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.562989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 792.565629] Lustre: Skipped 1 previous similar message [ 792.584437] Lustre: Skipped 1 previous similar message [ 797.665336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 797.681700] Lustre: Skipped 1 previous similar message [ 802.789642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 802.794089] Lustre: Skipped 1 previous similar message [ 804.320325] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 804.647853] Lustre: server umount lustre-MDT0000 complete [ 807.909212] LustreError: 27227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 807.939742] LustreError: 27227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 809.218901] LustreError: 27509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442755 with bad export cookie 16433195808081538871 [ 809.230965] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 809.237403] LustreError: 27509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 809.558704] Lustre: server umount lustre-MDT0001 complete [ 822.087328] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 825.065299] Key type lgssc unregistered [ 825.368849] LNet: 28265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.386484] LNetError: 28265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.408777] LNet: Removed LNI 192.168.204.113@tcp [ 826.279652] Key type .llcrypt unregistered [ 826.281529] Key type ._llcrypt unregistered [ 854.368391] Key type ._llcrypt registered [ 854.371945] Key type .llcrypt registered [ 854.472985] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 855.735858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 855.805162] alg: No test for adler32 (adler32-zlib) [ 856.981625] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 857.294955] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 858.992181] Key type lgssc registered [ 860.604672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 870.197843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 870.219353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.694992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 876.785502] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 882.154164] LustreError: 29678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 882.176136] LustreError: 29678:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 886.833827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.252607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 892.150234] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 900.209597] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 907.163610] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 921.672562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 922.133891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.036886] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 937.749668] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 941.629932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 951.784735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.789716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 951.791932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 951.814296] Lustre: Skipped 1 previous similar message [ 951.840292] LustreError: Skipped 1 previous similar message [ 955.495103] Lustre: server umount lustre-OST0000 complete [ 962.019956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.029326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 962.047634] Lustre: Skipped 1 previous similar message [ 962.063838] Lustre: Skipped 2 previous similar messages [ 967.138547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 967.149750] Lustre: Skipped 1 previous similar message [ 971.244467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 971.256988] Lustre: Skipped 1 previous similar message [ 974.304266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 974.740886] Lustre: server umount lustre-MDT0000 complete [ 976.356951] LustreError: 29678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.369750] LustreError: 29678:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 979.322422] LustreError: 29658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442926 with bad export cookie 1264145257642529594 [ 979.333728] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 979.335502] LustreError: 29658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 979.849674] Lustre: server umount lustre-MDT0001 complete [ 993.898960] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 997.086761] Key type lgssc unregistered [ 997.526565] LNet: 33563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.543148] LNetError: 33563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.582174] LNet: Removed LNI 192.168.204.113@tcp [ 998.899371] Key type .llcrypt unregistered [ 998.902451] Key type ._llcrypt unregistered [ 1025.234435] Key type ._llcrypt registered [ 1025.239717] Key type .llcrypt registered [ 1025.352345] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1026.419571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1026.574119] alg: No test for adler32 (adler32-zlib) [ 1027.602285] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1027.801443] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1029.448270] Key type lgssc registered [ 1030.929930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1040.468805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1040.495108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.026632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1046.747815] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1052.132960] LustreError: 34977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1052.156131] LustreError: 34977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1057.258874] LustreError: 34976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1057.422976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1057.877712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1063.801424] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1073.022178] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1079.236804] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1086.682109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1087.202420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1094.181070] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1102.471714] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1111.808656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1122.276260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1122.283613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.300286] Lustre: Skipped 1 previous similar message [ 1122.301731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1122.329946] Lustre: Skipped 1 previous similar message [ 1127.401879] LustreError: 36799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.426533] LustreError: 36799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1127.630433] Lustre: server umount lustre-OST0000 complete [ 1134.564806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1134.575810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.599124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.634332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.636876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.758057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.770208] Lustre: Skipped 2 previous similar messages [ 1147.360302] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1147.857746] Lustre: server umount lustre-MDT0000 complete [ 1147.885784] LustreError: 34972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1147.929566] LustreError: 34972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1152.484803] LustreError: 34959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443099 with bad export cookie 1554872109538121406 [ 1152.496306] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1152.505924] LustreError: 34959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1152.841665] Lustre: server umount lustre-MDT0001 complete [ 1166.887755] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1170.844984] Key type lgssc unregistered [ 1171.238853] LNet: 38539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.258970] LNetError: 38539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.292198] LNet: Removed LNI 192.168.204.113@tcp [ 1172.475733] Key type .llcrypt unregistered [ 1172.482112] Key type ._llcrypt unregistered [ 1200.958239] Key type ._llcrypt registered [ 1200.960870] Key type .llcrypt registered [ 1201.100477] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1203.288866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1203.365062] alg: No test for adler32 (adler32-zlib) [ 1204.492829] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1204.791687] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1206.528181] Key type lgssc registered [ 1207.832130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1217.577870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1217.628495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1219.506892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.849310] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1229.796593] LustreError: 39955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1229.818754] LustreError: 39955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1234.921731] LustreError: 39956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1236.872733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.298409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1243.268595] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1253.121926] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1261.336618] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1270.543849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1270.954233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.843786] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1286.940039] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1289.400550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1298.917495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1298.927282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.966500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1299.944185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.946414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1299.989787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1300.002311] Lustre: Skipped 2 previous similar messages [ 1304.228105] Lustre: lustre-OST0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 1305.128561] Lustre: server umount lustre-OST0000 complete [ 1329.180561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1329.209375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1329.223059] Lustre: Skipped 1 previous similar message [ 1334.244496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1334.258508] Lustre: Skipped 1 previous similar message [ 1335.782988] Lustre: server umount lustre-MDT0000 complete [ 1339.365227] LustreError: 39950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.385309] LustreError: 39950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1340.720951] LustreError: 39935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443287 with bad export cookie 10908643502501530955 [ 1340.724669] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1340.731742] LustreError: 39935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1355.234027] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1355.948786] Lustre: server umount lustre-MDT0001 complete [ 1370.215674] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1374.944433] Key type lgssc unregistered [ 1375.349517] LNet: 43675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1375.366530] LNetError: 43675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1375.400140] LNet: Removed LNI 192.168.204.113@tcp [ 1376.658771] Key type .llcrypt unregistered [ 1376.663627] Key type ._llcrypt unregistered [ 1405.401585] Key type ._llcrypt registered [ 1405.404456] Key type .llcrypt registered [ 1405.551831] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1406.855707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1407.032103] alg: No test for adler32 (adler32-zlib) [ 1408.248768] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1408.616787] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1410.392331] Key type lgssc registered [ 1412.115539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1420.822119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1420.847882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1422.207716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1426.969801] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1431.538106] LustreError: 45091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1431.557695] LustreError: 45091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1436.423322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.643647] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1436.781815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1442.168291] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1450.243813] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1457.211302] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1465.793602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1466.100059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1467.159916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1473.084200] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1483.165337] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1491.945426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1491.962593] Lustre: Skipped 1 previous similar message [ 1491.962804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1491.985555] Lustre: Skipped 2 previous similar messages [ 1495.213884] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 1496.624782] Lustre: server umount lustre-MDT0000 complete [ 1497.062687] LustreError: 46079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1497.097993] LustreError: 46079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1500.319711] LustreError: 45087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.184166] LustreError: 45072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443447 with bad export cookie 12183259030844529246 [ 1501.200821] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1501.204179] LustreError: 45072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1502.180636] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1502.205422] Lustre: Skipped 1 previous similar message [ 1502.216440] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1505.442924] LustreError: 45086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.471172] LustreError: 45086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1506.220273] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1506.228480] Lustre: Skipped 1 previous similar message [ 1507.909372] Lustre: server umount lustre-MDT0001 complete [ 1530.740526] Lustre: server umount lustre-OST0000 complete [ 1551.179403] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1553.956299] Key type lgssc unregistered [ 1554.299554] LNet: 48959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.309369] LNetError: 48959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.339643] LNet: Removed LNI 192.168.204.113@tcp [ 1555.262160] Key type .llcrypt unregistered [ 1555.264660] Key type ._llcrypt unregistered [ 1582.929743] Key type ._llcrypt registered [ 1582.931968] Key type .llcrypt registered [ 1583.086398] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1584.113415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1584.358501] alg: No test for adler32 (adler32-zlib) [ 1585.377372] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1585.705078] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1587.456502] Key type lgssc registered [ 1588.718862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1597.481051] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1597.518643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1613.089937] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1618.208401] LustreError: 50345:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.113@tcp: failed processing log, type 4: rc = -110 [ 1644.899241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1651.480844] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1660.072101] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1826.042345] Lustre: server umount lustre-OST0000 complete [ 1844.341170] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1847.497109] Key type lgssc unregistered [ 1847.848163] LNet: 52024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.858113] LNetError: 52024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.880703] LNet: Removed LNI 192.168.204.113@tcp [ 1848.881436] Key type .llcrypt unregistered [ 1848.885214] Key type ._llcrypt unregistered [ 1873.429802] Key type ._llcrypt registered [ 1873.432264] Key type .llcrypt registered [ 1873.568258] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1874.731993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1874.854945] alg: No test for adler32 (adler32-zlib) [ 1875.946520] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1876.234563] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1877.995877] Key type lgssc registered [ 1879.345786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1887.724784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1887.743440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1889.198844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.694803] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1898.476558] LustreError: 53441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1898.514845] LustreError: 53441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1903.592381] LustreError: 53440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1903.771591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1904.103160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1909.531574] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1918.345934] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1925.056395] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1933.496316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1934.023390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.487056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1943.507440] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1952.488926] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1958.891870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1958.892937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.921571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1958.929938] Lustre: Skipped 1 previous similar message [ 1958.952832] Lustre: Skipped 1 previous similar message [ 1962.114142] Lustre: server umount lustre-OST0000 complete [ 1967.010426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.026881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1967.033607] Lustre: Skipped 1 previous similar message [ 1969.058410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1969.068774] Lustre: Skipped 1 previous similar message [ 1973.215856] Lustre: server umount lustre-MDT0000 complete [ 1974.244894] LustreError: 53440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.270651] LustreError: 53440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1977.284268] LustreError: 53420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443924 with bad export cookie 117703942909417446 [ 1977.289218] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1977.298639] LustreError: 53420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1977.730719] Lustre: server umount lustre-MDT0001 complete [ 1991.010099] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1994.594285] Key type lgssc unregistered [ 1994.936034] LNet: 56976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1994.955407] LNetError: 56976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1994.971365] LNet: Removed LNI 192.168.204.113@tcp [ 1996.034787] Key type .llcrypt unregistered [ 1996.039821] Key type ._llcrypt unregistered [ 2023.191409] Key type ._llcrypt registered [ 2023.194393] Key type .llcrypt registered [ 2023.319641] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2024.498216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2024.637513] alg: No test for adler32 (adler32-zlib) [ 2025.826457] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2026.198962] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2027.921477] Key type lgssc registered [ 2029.227561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2037.756656] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2037.770562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2053.344898] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2058.467146] LustreError: 58374:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.113@tcp: failed processing log, type 4: rc = -110 [ 2085.216682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2092.196791] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2100.660853] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2109.233994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.964858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2110.087851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2114.897688] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2115.043556] LustreError: 58399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.102119] LustreError: 58399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2120.167437] LustreError: 58398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.205288] LustreError: 58398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2125.283069] LustreError: 58399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.300248] LustreError: 58399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2127.202546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.683147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2134.298875] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2144.950203] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2152.137902] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2156.001728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2156.007240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.020662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2158.566242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.569126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2163.682206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2163.693834] Lustre: Skipped 2 previous similar messages [ 2168.807125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2168.814722] Lustre: Skipped 1 previous similar message [ 2169.312207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2169.504380] Lustre: server umount lustre-OST0000 complete [ 2194.404673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2194.408197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.418420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2194.442390] Lustre: Skipped 1 previous similar message [ 2203.617130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2203.623400] Lustre: Skipped 3 previous similar messages [ 2207.200495] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2207.665262] Lustre: server umount lustre-MDT0000 complete [ 2208.750216] LustreError: 59414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2208.773243] LustreError: 59414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2211.455072] LustreError: 58381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444158 with bad export cookie 12649346927157168885 [ 2211.465494] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2211.476105] LustreError: 58381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2218.110845] Lustre: server umount lustre-MDT0001 complete [ 2228.307667] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2231.296756] Key type lgssc unregistered [ 2231.678778] LNet: 62139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.689786] LNetError: 62139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2231.706949] LNet: Removed LNI 192.168.204.113@tcp [ 2232.824400] Key type .llcrypt unregistered [ 2232.829269] Key type ._llcrypt unregistered [ 2259.030736] Key type ._llcrypt registered [ 2259.032391] Key type .llcrypt registered [ 2259.166397] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2260.560697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2260.602414] alg: No test for adler32 (adler32-zlib) [ 2261.600966] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2261.901585] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2263.600200] Key type lgssc registered [ 2264.808748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2272.676383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2272.708984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2274.143452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.681919] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2283.494928] LustreError: 63551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2283.516473] LustreError: 63551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2288.016236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.444166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2293.671609] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2301.876486] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2309.382475] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2318.088073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2318.322923] LustreError: 65375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.536618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.962512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2326.081949] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2334.419769] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2354.069683] Lustre: server umount lustre-OST0000 complete [ 2359.778172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.780942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2359.790312] Lustre: Skipped 1 previous similar message [ 2359.807612] Lustre: Skipped 1 previous similar message [ 2364.905419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2364.916775] Lustre: Skipped 1 previous similar message [ 2369.099221] Lustre: server umount lustre-MDT0000 complete [ 2370.017938] LustreError: 63552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.042794] LustreError: 63552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2371.041555] Lustre: 62865:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444301/real 1788444301] req@000000003bce4b6c x1875319673877120/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788444317 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2371.111036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.065787] LustreError: 63533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444320 with bad export cookie 13285300305654287661 [ 2374.074811] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2374.079931] LustreError: 63533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2374.901962] Lustre: server umount lustre-MDT0001 complete [ 2388.041523] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2392.061151] Key type lgssc unregistered [ 2392.354750] LNet: 67116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2392.366794] LNetError: 67116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2392.389898] LNet: Removed LNI 192.168.204.113@tcp [ 2393.260187] Key type .llcrypt unregistered [ 2393.262770] Key type ._llcrypt unregistered [ 2427.679806] Key type ._llcrypt registered [ 2427.683290] Key type .llcrypt registered [ 2427.863763] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2428.975411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2429.043875] alg: No test for adler32 (adler32-zlib) [ 2430.307604] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2430.863552] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2432.768231] Key type lgssc registered [ 2434.656735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2446.406469] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 10:06:30 (1788444390) [ 2453.385028] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2453.403709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2454.901582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2460.021300] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2464.232736] LustreError: 68819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.258927] LustreError: 68819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2469.345466] LustreError: 68818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.629227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2470.070799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2476.052467] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2485.320429] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2491.635936] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2500.045508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2500.607799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2508.019371] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2516.195550] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2525.192692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2533.589780] Lustre: Failing over lustre-MDT0000 [ 2534.400106] Lustre: server umount lustre-MDT0000 complete [ 2535.398722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.412077] LustreError: 68813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.435339] Lustre: Skipped 1 previous similar message [ 2535.474935] LustreError: 68813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2537.643552] LustreError: 68814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.763992] LustreError: 71320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.797592] LustreError: 71320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2547.277560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2547.489883] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2547.750649] Lustre: *** cfs_fail_loc=135, val=0*** [ 2547.861230] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.13@tcp (not set up) [ 2547.925824] LustreError: 71938:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2547.952395] LustreError: 71938:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.113@tcp: cfg command failed: rc = -2 [ 2547.980148] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2547.980148] [ 2547.987245] LustreError: MGC192.168.204.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2548.013656] LustreError: 71926:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2548.046294] LustreError: 71926:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2548.072258] LustreError: 71926:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2548.088304] LustreError: 71926:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2548.133608] Lustre: server umount lustre-MDT0000 complete [ 2548.135799] LustreError: 71926:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2552.991810] LustreError: 68814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.046018] LustreError: 68814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2556.926090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2557.251826] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2557.693594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.786600] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2558.105670] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2562.936121] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2563.088723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2563.137039] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2563.212105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2573.288347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.296372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2573.308528] Lustre: Skipped 2 previous similar messages [ 2578.356272] Lustre: server umount lustre-OST0000 complete [ 2583.528103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2583.528661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2583.532603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2583.532615] Lustre: Skipped 1 previous similar message [ 2583.554079] Lustre: Skipped 1 previous similar message [ 2588.641936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2588.656069] Lustre: Skipped 2 previous similar messages [ 2589.779585] Lustre: server umount lustre-MDT0000 complete [ 2593.786042] LustreError: 69771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.813571] LustreError: 69771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2595.072610] LustreError: 69554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444541 with bad export cookie 18196827890771228245 [ 2595.084551] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2595.099285] LustreError: 69554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2595.671869] Lustre: server umount lustre-MDT0001 complete [ 2609.735714] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2613.835736] Key type lgssc unregistered [ 2614.268096] LNet: 73723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2614.288713] LNetError: 73723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2615.334867] LNet: Removed LNI 192.168.204.113@tcp [ 2616.729652] Key type .llcrypt unregistered [ 2616.733344] Key type ._llcrypt unregistered [ 2643.860219] Key type ._llcrypt registered [ 2643.862666] Key type .llcrypt registered [ 2643.964597] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2645.459943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2645.483186] alg: No test for adler32 (adler32-zlib) [ 2646.470531] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2646.743645] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2648.416176] Key type lgssc registered [ 2649.684451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2657.879593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2657.902248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.546709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.751741] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2670.064783] LustreError: 75138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2670.097447] LustreError: 75138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2673.813047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.269240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2679.406322] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2688.080088] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2694.713576] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2703.876577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2704.377829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2705.403693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2712.018898] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2722.244277] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2728.267051] Lustre: Failing over lustre-MDT0000 [ 2728.602732] Lustre: server umount lustre-MDT0000 complete [ 2728.944509] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.952851] LustreError: 75139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.965758] Lustre: Skipped 2 previous similar messages [ 2729.010766] LustreError: 75139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2734.056033] LustreError: 75138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.107091] LustreError: 75138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2739.174074] LustreError: 75133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.215926] LustreError: 75133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2742.413372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.589171] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2742.878856] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2743.088215] LustreError: 78229:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2743.108396] LustreError: 78229:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.113@tcp: cfg command failed: rc = -9 [ 2743.121191] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2743.121191] [ 2743.144633] LustreError: MGC192.168.204.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2743.168824] LustreError: 78217:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2743.186088] LustreError: 78217:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2743.199409] LustreError: 78217:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2743.208052] LustreError: 78217:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2743.242715] Lustre: server umount lustre-MDT0000 complete [ 2743.244843] LustreError: 78217:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2744.295606] LustreError: 75139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.323965] LustreError: 75139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2751.607717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.724938] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2751.733801] LustreError: 75130:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2752.101929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2752.169225] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2754.722698] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2756.589039] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2757.096773] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2757.128781] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2757.204520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2767.332397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.336109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2767.390827] Lustre: Skipped 1 previous similar message [ 2767.427860] Lustre: Skipped 1 previous similar message [ 2772.459478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2772.466781] Lustre: Skipped 1 previous similar message [ 2772.924886] Lustre: server umount lustre-OST0000 complete [ 2777.570212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2777.572131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2777.595227] Lustre: Skipped 1 previous similar message [ 2777.595545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2777.638223] Lustre: Skipped 1 previous similar message [ 2782.689110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.699708] Lustre: Skipped 1 previous similar message [ 2787.812910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2791.392205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2791.629660] Lustre: server umount lustre-MDT0000 complete [ 2792.932281] LustreError: 75134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.948786] LustreError: 75134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2795.683096] LustreError: 75118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444742 with bad export cookie 17857080271542082708 [ 2795.693278] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2795.698481] LustreError: 75118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2796.135756] Lustre: server umount lustre-MDT0001 complete [ 2808.876911] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2812.026759] Key type lgssc unregistered [ 2812.369899] LNet: 80014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2812.394864] LNetError: 80014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2812.421840] LNet: Removed LNI 192.168.204.113@tcp [ 2813.777288] Key type .llcrypt unregistered [ 2813.780452] Key type ._llcrypt unregistered [ 2827.708560] Key type ._llcrypt registered [ 2827.711900] Key type .llcrypt registered [ 2827.796919] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2828.155738] Key type .llcrypt unregistered [ 2828.159699] Key type ._llcrypt unregistered [ 2855.268164] Key type ._llcrypt registered [ 2855.272571] Key type .llcrypt registered [ 2855.408272] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2856.784530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2856.866298] alg: No test for adler32 (adler32-zlib) [ 2858.022022] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2858.277494] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2860.032313] Key type lgssc registered [ 2861.374726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2870.218531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2870.229175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2871.829772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.238711] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2882.025224] LustreError: 81920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.054516] LustreError: 81920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2887.137355] LustreError: 81921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.285520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.744638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2897.407215] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2906.273973] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2913.691739] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2921.652926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2921.949461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2929.855691] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2938.252503] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2941.501575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2955.751575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2955.765959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2955.806759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2956.773971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2956.776662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2956.793967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.592776] Lustre: server umount lustre-OST0000 complete [ 2967.020946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.048319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2967.059053] Lustre: Skipped 1 previous similar message [ 2967.090156] Lustre: Skipped 2 previous similar messages [ 2971.382391] Lustre: server umount lustre-MDT0000 complete [ 2972.130616] LustreError: 83354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2972.163523] LustreError: 83354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2975.952661] LustreError: 81901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444922 with bad export cookie 16918909577978510902 [ 2975.958133] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2975.979326] LustreError: 81901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2976.257089] Lustre: server umount lustre-MDT0001 complete [ 2991.064811] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2994.276831] Key type lgssc unregistered [ 2994.615825] LNet: 85482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2994.623192] LNetError: 85482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2994.645282] LNet: Removed LNI 192.168.204.113@tcp [ 2996.278552] Key type .llcrypt unregistered [ 2996.280295] Key type ._llcrypt unregistered [ 3026.157918] Key type ._llcrypt registered [ 3026.160149] Key type .llcrypt registered [ 3026.334598] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3028.002393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3028.049041] alg: No test for adler32 (adler32-zlib) [ 3029.433334] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3029.987861] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3031.944222] Key type lgssc registered [ 3033.936487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3043.734419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3043.766346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3045.117973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3050.954661] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3054.577046] LustreError: 86897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.595552] LustreError: 86897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3059.717886] LustreError: 86898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3061.914359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.439304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3067.622555] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3078.352798] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3086.153095] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3095.128030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3095.426506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3102.785690] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3112.022376] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3114.873420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3128.300357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3128.311515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.336317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3130.340814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3130.341921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3130.372100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3131.869465] Lustre: server umount lustre-OST0000 complete [ 3139.049457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3139.066652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.097030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.104742] Lustre: Skipped 1 previous similar message [ 3142.589287] Lustre: server umount lustre-MDT0000 complete [ 3145.709415] LustreError: 86893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3145.741787] LustreError: 86893:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3147.109165] LustreError: 89700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445093 with bad export cookie 1539985876766634424 [ 3147.112206] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.117398] LustreError: 89700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3147.451318] Lustre: server umount lustre-MDT0001 complete [ 3161.280225] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 3165.122688] Key type lgssc unregistered [ 3165.560610] LNet: 90456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3165.580674] LNetError: 90456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3165.604563] LNet: Removed LNI 192.168.204.113@tcp [ 3166.649997] Key type .llcrypt unregistered [ 3166.652084] Key type ._llcrypt unregistered [ 3194.854230] Key type ._llcrypt registered [ 3194.858752] Key type .llcrypt registered [ 3195.015091] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3196.584579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3196.838388] alg: No test for adler32 (adler32-zlib) [ 3198.000887] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3198.514305] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3200.376172] Key type lgssc registered [ 3201.721674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3210.762932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3210.779973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3212.236436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.026343] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3221.479783] LustreError: 91867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3221.504439] LustreError: 91867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3226.599934] LustreError: 91866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3227.244597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3227.793206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3232.950917] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3240.804328] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3247.700780] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3257.014777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3257.512285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.203198] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3274.469850] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3282.146881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3298.793207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3298.803564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.836918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3302.883099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3302.885882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3302.893693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.800785] Lustre: server umount lustre-OST0000 complete [ 3309.538327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3309.543414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.565148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.573485] Lustre: Skipped 1 previous similar message [ 3313.121884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3313.124447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3318.242774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3318.252738] Lustre: Skipped 2 previous similar messages [ 3323.361705] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3323.614618] Lustre: server umount lustre-MDT0000 complete [ 3327.466226] LustreError: 91862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3327.487141] LustreError: 91862:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3327.534042] LustreError: 91847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445274 with bad export cookie 9543197480252406882 [ 3327.543707] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3327.567340] LustreError: 91847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3328.009712] Lustre: server umount lustre-MDT0001 complete [ 3338.904623] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 3342.251932] Key type lgssc unregistered [ 3342.730899] LNet: 95428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3342.740158] LNetError: 95428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3342.759658] LNet: Removed LNI 192.168.204.113@tcp [ 3343.834614] Key type .llcrypt unregistered [ 3343.837636] Key type ._llcrypt unregistered [ 3370.307935] Key type ._llcrypt registered [ 3370.311430] Key type .llcrypt registered [ 3370.525835] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3371.689480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3371.938107] alg: No test for adler32 (adler32-zlib) [ 3373.012674] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3373.190484] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3374.872363] Key type lgssc registered [ 3376.077347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3386.213256] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3386.237859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.954957] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3407.072318] LustreError: 96816:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.113@tcp: failed processing log, type 4: rc = -110 [ 3433.760468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3441.869403] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3451.459756] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3458.825372] Lustre: server umount lustre-OST0000 complete [ 3468.689254] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 10:23:33 (1788445413) [ 3476.761739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3477.282745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3482.012846] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3482.607315] LustreError: 98455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3482.642178] LustreError: 98455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3487.714458] LustreError: 98456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3491.387519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.892967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3497.077988] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3507.904274] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3516.778901] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3526.351909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3527.022630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3534.709460] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3543.113827] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3547.510361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3561.957814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3561.971342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3562.006583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3562.984923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3562.995372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3563.011397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3563.018949] Lustre: Skipped 1 previous similar message [ 3565.608396] Lustre: server umount lustre-OST0000 complete [ 3573.220068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3573.222306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3573.257937] Lustre: Skipped 1 previous similar message [ 3573.266413] Lustre: Skipped 1 previous similar message [ 3576.310800] Lustre: server umount lustre-MDT0000 complete [ 3578.338414] LustreError: 98450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3578.363169] LustreError: 98450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3580.915784] LustreError: 98437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445527 with bad export cookie 15945548627532438886 [ 3580.928187] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3580.935709] LustreError: 98437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3581.332166] Lustre: server umount lustre-MDT0001 complete [ 3592.886394] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 3596.391134] Key type lgssc unregistered [ 3596.691206] LNet: 102161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3596.702356] LNetError: 102161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3596.723782] LNet: Removed LNI 192.168.204.113@tcp [ 3597.590199] Key type .llcrypt unregistered [ 3597.592949] Key type ._llcrypt unregistered [ 3642.816156] Key type ._llcrypt registered [ 3642.817840] Key type .llcrypt registered [ 3642.934885] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3643.785944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3643.817489] alg: No test for adler32 (adler32-zlib) [ 3644.861311] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3645.116068] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3646.776206] Key type lgssc registered [ 3648.053777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3656.408443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3656.432219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.039572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3662.721878] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3667.428256] LustreError: 104348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3667.448581] LustreError: 104348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3672.548048] LustreError: 104347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3672.710945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3673.216068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3678.570049] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3688.227360] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3695.986632] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3704.727081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3705.246123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3712.044533] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3721.809799] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3727.812358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3743.209743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3743.217493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.265460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3747.382845] Lustre: server umount lustre-OST0000 complete [ 3753.443161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3753.451395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3753.480832] Lustre: Skipped 2 previous similar messages [ 3753.502372] Lustre: Skipped 2 previous similar messages [ 3758.564846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3758.576528] Lustre: Skipped 1 previous similar message [ 3763.686205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3763.694218] Lustre: Skipped 1 previous similar message [ 3766.241336] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3766.456539] Lustre: server umount lustre-MDT0000 complete [ 3768.803236] LustreError: 104348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.814036] LustreError: 104348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3770.781687] LustreError: 104327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445717 with bad export cookie 14695873084982755241 [ 3770.787169] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3770.796728] LustreError: 104327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3771.124982] Lustre: server umount lustre-MDT0001 complete [ 3783.986975] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 3787.479907] Key type lgssc unregistered [ 3787.888343] LNet: 107911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3787.913596] LNetError: 107911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3787.932671] LNet: Removed LNI 192.168.204.113@tcp [ 3789.011058] Key type .llcrypt unregistered [ 3789.013208] Key type ._llcrypt unregistered [ 3810.526547] Key type ._llcrypt registered [ 3810.529394] Key type .llcrypt registered [ 3810.656216] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3812.009901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3812.051649] alg: No test for adler32 (adler32-zlib) [ 3813.354181] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3813.714901] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3815.408191] Key type lgssc registered [ 3817.025169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3826.447158] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3826.477702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3842.082968] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3847.200545] LustreError: 109152:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.113@tcp: failed processing log, type 4: rc = -110 [ 3873.888222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3881.940111] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3891.700841] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3900.718787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3901.081111] LustreError: MGC192.168.204.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3901.098358] LustreError: 110177:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3901.113323] LustreError: 110177:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3901.125295] LustreError: 110177:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3901.209805] Lustre: server umount lustre-MDT0000 complete [ 3901.212114] LustreError: 110177:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3922.912258] Lustre: 108490:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445853/real 1788445853] req@00000000f36d604e x1875321301241216/t0(0) o400->MGC192.168.204.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788445869 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3922.958395] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3922.975091] LustreError: 109165:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3943.545242] Lustre: server umount lustre-OST0000 complete [ 3954.054391] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 3965.028866] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4011.497175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4011.895657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4011.951528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4012.109545] Lustre: lustre-MDT0000: new disk, initializing [ 4012.208921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4012.239268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4017.109514] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4028.016845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4028.124421] Lustre: 112972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4028.186252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4028.190447] Lustre: Skipped 1 previous similar message [ 4028.264328] Lustre: lustre-MDT0001: new disk, initializing [ 4028.390702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4028.474634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4028.499116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4032.691494] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4042.354524] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4048.482073] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4055.860090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4056.150785] Lustre: lustre-OST0000: new disk, initializing [ 4056.157047] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4056.166186] Lustre: 114943:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4056.260803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4061.776127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4061.782028] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4061.904118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4062.272576] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4072.013547] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4077.177588] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4077.410662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4083.034745] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4083.221963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4087.785796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4087.787377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4087.788389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4087.849383] Lustre: Skipped 1 previous similar message [ 4091.310733] Lustre: server umount lustre-OST0000 complete [ 4098.019553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4098.055662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4098.070854] Lustre: Skipped 2 previous similar messages [ 4101.624110] Lustre: server umount lustre-MDT0000 complete [ 4106.936742] LustreError: 114953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446053 with bad export cookie 2884844243569186665 [ 4106.957982] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4106.975673] LustreError: 114953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4107.071484] LustreError: 108489:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 4107.459625] Lustre: server umount lustre-MDT0001 complete [ 4123.287497] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:34:27 (1788446067) [ 4127.924543] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4145.386796] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4155.224617] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4195.886759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4196.221784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4196.258988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4196.369924] Lustre: lustre-MDT0000: new disk, initializing [ 4196.499718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4196.532632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4201.119204] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4212.600385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4212.747755] Lustre: 119846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4212.804794] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4212.807969] Lustre: Skipped 1 previous similar message [ 4212.860187] Lustre: lustre-MDT0001: new disk, initializing [ 4212.947701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4212.989327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4212.999228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4217.661236] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4227.924690] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4234.552657] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4242.073830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4242.376829] Lustre: lustre-OST0000: new disk, initializing [ 4242.381365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4242.389896] Lustre: 121814:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4242.499032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4247.714606] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4248.649109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4248.659901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4248.759216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4256.842195] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4261.467678] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4261.707938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4266.026725] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4266.261829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4269.553732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4269.557833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4269.579293] Lustre: Skipped 1 previous similar message [ 4269.591099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4269.595041] LustreError: Skipped 1 previous similar message [ 4274.664713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4274.674575] Lustre: Skipped 2 previous similar messages [ 4279.778557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4279.784768] Lustre: Skipped 1 previous similar message [ 4282.337042] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4282.487303] Lustre: server umount lustre-OST0000 complete [ 4290.022079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4290.029167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4290.058608] Lustre: Skipped 1 previous similar message [ 4290.071390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4290.086041] Lustre: Skipped 1 previous similar message [ 4301.280512] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4301.569414] Lustre: server umount lustre-MDT0000 complete [ 4305.353055] LustreError: 119840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446252 with bad export cookie 2884844243569187554 [ 4305.358839] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4305.363296] LustreError: 119840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4305.840168] Lustre: server umount lustre-MDT0001 complete [ 4314.633583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.998940] LustreError: 124035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.017052] LustreError: 124035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4315.071519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4319.498623] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4320.228085] LustreError: 124036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.259483] LustreError: 124036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4324.328935] LustreError: 124035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4328.231247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4333.191941] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4340.777572] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4346.080848] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4352.825219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4359.164349] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4366.986243] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4373.246682] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4380.134317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4380.135616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4380.139326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4380.139818] Lustre: Skipped 4 previous similar messages [ 4380.146346] LustreError: Skipped 1 previous similar message [ 4380.153713] Lustre: Skipped 2 previous similar messages [ 4383.012931] Lustre: server umount lustre-OST0000 complete [ 4389.858386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4389.871693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4392.891985] Lustre: server umount lustre-MDT0000 complete [ 4395.490880] LustreError: 124035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4395.512180] LustreError: 124035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4396.634840] LustreError: 124016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446343 with bad export cookie 2884844243569188429 [ 4396.662245] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4396.662886] LustreError: 124016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4397.056514] Lustre: server umount lustre-MDT0001 complete [ 4408.942223] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4411.672569] Key type lgssc unregistered [ 4412.060327] LNet: 127667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4412.064210] LNetError: 127667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4412.086287] LNet: Removed LNI 192.168.204.113@tcp [ 4412.884070] Key type .llcrypt unregistered [ 4412.886688] Key type ._llcrypt unregistered [ 4430.292991] Key type ._llcrypt registered [ 4430.298325] Key type .llcrypt registered [ 4430.421613] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4446.912704] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4448.226215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4448.327025] alg: No test for adler32 (adler32-zlib) [ 4449.411949] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4449.687373] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4451.432379] Key type lgssc registered [ 4452.644773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4485.970210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4485.990901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4487.161683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4487.182086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4487.245851] Lustre: lustre-MDT0000: new disk, initializing [ 4487.312192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4487.330454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4491.213883] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4501.191750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4501.331329] Lustre: 130822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4501.444479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4501.452843] Lustre: Skipped 1 previous similar message [ 4501.528314] Lustre: lustre-MDT0001: new disk, initializing [ 4501.623618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4501.673879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4501.689722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4506.704417] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4516.425635] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4522.865518] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4530.311323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4530.559350] Lustre: lustre-OST0000: new disk, initializing [ 4530.562748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4530.572560] Lustre: 132794:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4530.677595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4537.282142] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4540.477854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4540.494531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4540.617780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4547.074225] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4550.914781] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4551.093440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4555.940384] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4556.210421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4561.383432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4561.383581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4561.402956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4561.406714] LustreError: Skipped 1 previous similar message [ 4564.759658] Lustre: server umount lustre-OST0000 complete [ 4571.621019] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4571.625485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4571.639045] Lustre: Skipped 2 previous similar messages [ 4571.658888] Lustre: Skipped 2 previous similar messages [ 4574.700026] Lustre: server umount lustre-MDT0000 complete [ 4577.918957] LustreError: 130815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446524 with bad export cookie 4328900283613337549 [ 4577.921965] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4577.937387] LustreError: 130815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4578.243530] Lustre: server umount lustre-MDT0001 complete [ 4587.076543] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:42:12 (1788446532) [ 4595.008721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4595.493840] LustreError: 135204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4595.527800] LustreError: 135204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4595.606420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4600.707521] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4600.805306] LustreError: 135205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.930704] LustreError: 135204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.643752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.908284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4615.558607] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4624.353613] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4631.235674] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4635.432482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4635.437181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4635.448581] Lustre: Skipped 1 previous similar message [ 4639.724607] Lustre: server umount lustre-MDT0000 complete [ 4640.738507] LustreError: 135199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4640.770916] LustreError: 135199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4643.330571] LustreError: 135186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446590 with bad export cookie 4328900283613338431 [ 4643.347990] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4643.600815] Lustre: server umount lustre-MDT0001 complete [ 4650.995560] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:43:16 (1788446596) [ 4657.484447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4673.056747] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4678.177900] LustreError: 137619:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.113@tcp: failed processing log, type 4: rc = -110 [ 4703.971070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4709.288523] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4716.293440] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4718.163090] Lustre: server umount lustre-OST0000 complete [ 4726.292650] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:44:31 (1788446671) [ 4732.702595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4733.249661] LustreError: 139065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4733.336667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4738.276265] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4743.650879] LustreError: 139066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4743.672104] LustreError: 139066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4747.431146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4752.562304] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4759.836858] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4766.107177] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4773.745252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4774.083949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4774.092599] Lustre: Skipped 1 previous similar message [ 4780.216674] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4787.484711] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4803.042817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4803.048820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4803.063059] Lustre: Skipped 1 previous similar message [ 4803.089428] Lustre: Skipped 3 previous similar messages [ 4808.174776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4808.184868] Lustre: Skipped 2 previous similar messages [ 4815.328482] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4815.447200] Lustre: server umount lustre-MDT0000 complete [ 4818.402686] LustreError: 139065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4818.416177] LustreError: 139065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4818.938912] LustreError: 139047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446765 with bad export cookie 4328900283613339096 [ 4818.950837] LustreError: 139047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4818.954111] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4819.219556] Lustre: server umount lustre-MDT0001 complete [ 4833.128788] Lustre: server umount lustre-OST0000 complete [ 4841.575622] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:46:26 (1788446786) [ 4847.886987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4848.433281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4852.965421] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4853.731172] LustreError: 142739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4853.748728] LustreError: 142739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4862.662041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.257631] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4874.633304] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4880.965194] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4887.902552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4893.363034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4893.949561] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4901.887134] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4906.819348] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4907.073979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4911.535910] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4911.742071] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4914.145869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4914.156719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4914.172309] Lustre: Skipped 1 previous similar message [ 4914.181193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4914.187060] Lustre: Skipped 3 previous similar messages [ 4918.250444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4919.910789] Lustre: server umount lustre-OST0000 complete [ 4923.361671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4923.379354] Lustre: Skipped 2 previous similar messages [ 4929.578822] Lustre: server umount lustre-MDT0000 complete [ 4933.255647] LustreError: 142720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446880 with bad export cookie 4328900283613340167 [ 4933.261481] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4933.270824] LustreError: 142720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4933.719233] Lustre: server umount lustre-MDT0001 complete [ 4941.431825] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:48:06 (1788446886) [ 4947.635727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4963.169320] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4968.288479] LustreError: 146899:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.113@tcp: failed processing log, type 4: rc = -110 [ 4994.016319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4994.023210] Lustre: Skipped 2 previous similar messages [ 4999.588771] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5006.949242] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5014.742690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5015.225678] LustreError: 146924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5015.256277] LustreError: 146924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5015.423725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5021.078600] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5030.401711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5034.897500] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5041.968251] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5047.192412] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5051.476867] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5051.691828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5055.699772] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5055.878124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5061.605727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5061.607675] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5061.622614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5061.622622] Lustre: Skipped 6 previous similar messages [ 5061.656222] Lustre: Skipped 2 previous similar messages [ 5072.352940] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5072.505717] Lustre: server umount lustre-MDT0000 complete [ 5075.890037] LustreError: 146907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447022 with bad export cookie 4328900283613341189 [ 5075.892035] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5075.905579] LustreError: 146907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5076.254557] Lustre: server umount lustre-MDT0001 complete [ 5089.933284] Lustre: server umount lustre-OST0000 complete [ 5097.790901] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:50:43 (1788447043) [ 5103.931185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5119.520708] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5124.640447] LustreError: 151097:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.113@tcp: failed processing log, type 4: rc = -110 [ 5156.261231] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5165.105801] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5172.029587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5172.438438] LustreError: 151122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5172.456660] LustreError: 151122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5172.577625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5177.061100] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5185.172252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5189.571321] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5196.988344] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5203.325303] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5209.521370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5209.692372] Lustre: lustre-OST0001: new disk, initializing [ 5209.696837] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5209.707694] Lustre: 153969:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5215.943343] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5219.402196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5219.418961] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5219.532384] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5225.481973] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5230.794526] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5230.973486] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5235.359675] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5235.654366] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5239.268386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5239.278352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5239.289771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5239.295752] Lustre: Skipped 8 previous similar messages [ 5243.558397] Lustre: server umount lustre-OST0000 complete [ 5250.532838] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5250.550308] LustreError: Skipped 1 previous similar message [ 5253.803415] Lustre: server umount lustre-OST0001 complete [ 5263.951435] Lustre: server umount lustre-MDT0000 complete [ 5267.826602] LustreError: 153968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447214 with bad export cookie 4328900283613342141 [ 5267.830446] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5267.843074] LustreError: 153968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5268.226237] Lustre: server umount lustre-MDT0001 complete [ 5291.432085] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:53:56 (1788447236) [ 5293.008803] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5294.511047] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5296.085282] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:54:01 (1788447241) [ 5302.293340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5302.500937] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5302.523465] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5302.669949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5302.674460] Lustre: Skipped 6 previous similar messages [ 5306.844845] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5315.161630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5315.277214] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5315.317679] Lustre: 157529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5320.591787] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5328.052868] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5333.278916] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5339.312572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5339.506038] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5345.119253] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5346.933323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5352.396308] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5356.878086] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5357.120975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5361.614080] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5361.857727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5367.266712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5367.282945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.295111] Lustre: Skipped 5 previous similar messages [ 5367.297880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5367.300312] Lustre: Skipped 6 previous similar messages [ 5369.717471] Lustre: server umount lustre-OST0000 complete [ 5390.276284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5396.226700] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5402.974254] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5407.982746] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.993736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.996665] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5408.029323] Lustre: Skipped 1 previous similar message [ 5408.106983] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5408.351160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5412.424448] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5412.624324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5416.881651] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8fb147198800.ost_server_uuid 50 [ 5418.176381] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8fb147198800.ost_server_uuid in FULL state after 0 sec [ 5418.336885] LustreError: 157536:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.13@tcp) returned error from blocking AST (req@00000000ff166a9d x1875321968388480 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000005f058ff1/0x3c1357df58dc7317 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.13@tcp remote: 0xbd410773512b062c expref: 5 pid: 157535 timeout: 5518 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5418.373481] LustreError: lustre-MDT0000: A client on nid 192.168.204.13@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5418.382643] LustreError: 157524:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.13@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000005f058ff1/0x3c1357df58dc7317 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.13@tcp remote: 0xbd410773512b062c expref: 6 pid: 157535 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5423.592632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5423.603306] LustreError: Skipped 1 previous similar message [ 5428.042880] Lustre: server umount lustre-OST0000 complete [ 5437.510102] Lustre: server umount lustre-MDT0000 complete [ 5438.952662] LustreError: 157535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5439.002552] LustreError: 157535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 5440.871751] LustreError: 157521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447387 with bad export cookie 4328900283613343324 [ 5440.873977] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5440.880938] LustreError: 157521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5455.328135] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5455.558489] Lustre: server umount lustre-MDT0001 complete [ 5465.497666] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 5468.260306] Key type lgssc unregistered [ 5468.547512] LNet: 163642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5468.551918] LNetError: 163642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5469.614152] LNet: Removed LNI 192.168.204.113@tcp [ 5470.308750] Key type .llcrypt unregistered [ 5470.311062] Key type ._llcrypt unregistered [ 5493.779927] Key type ._llcrypt registered [ 5493.782415] Key type .llcrypt registered [ 5493.880646] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5495.210309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5495.254314] alg: No test for adler32 (adler32-zlib) [ 5496.264723] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5496.715297] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 5498.400273] Key type lgssc registered [ 5500.125611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5508.516910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5508.571254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5510.008502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5514.926039] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5519.332218] LustreError: 165058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5519.392974] LustreError: 165058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5524.454689] LustreError: 165058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5524.741463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.259140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5530.692614] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5538.871649] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5545.646103] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5553.005477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5553.359790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5554.432086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5559.332386] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5567.465801] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5576.186273] Lustre: Failing over lustre-MDT0000 [ 5576.404987] Lustre: server umount lustre-MDT0000 complete [ 5576.675515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5576.688604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5576.712146] LustreError: 165057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5580.264462] LustreError: 165058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5580.270992] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.289603] LustreError: 165058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5580.314189] Lustre: Skipped 1 previous similar message [ 5585.380656] LustreError: 165054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5585.407203] LustreError: 165054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5588.448830] LustreError: 165037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.13@tcp arrived at 1788447535 with bad export cookie 3645386140624691522 [ 5595.460935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5595.588635] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5595.619954] LustreError: 165053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5595.635975] LustreError: 165053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5595.792942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5595.838861] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5600.201969] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5601.255480] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5601.259911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5602.394730] Lustre: lustre-MDT0000: Denying connection for new client 5d7813d0-5389-493e-8369-72380a4ea299 (at 192.168.204.13@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5610.977389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5610.990726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5611.001939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5616.614330] LustreError: 166948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5616.644797] LustreError: 166948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5616.741562] Lustre: server umount lustre-OST0000 complete [ 5620.893905] LustreError: 168788:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5620.914353] Lustre: 168137:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5620.920644] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5620.933165] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5620.942338] LustreError: 168137:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f6a5c1d0 x1875323066036224/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5620.956494] LustreError: 168137:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5620.965432] LustreError: 168137:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5620.972327] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 5621.568788] Lustre: server umount lustre-MDT0000 complete [ 5621.750324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5621.782884] Lustre: Skipped 1 previous similar message [ 5625.685962] LustreError: 165039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447572 with bad export cookie 3645386140624691627 [ 5625.714848] LustreError: 165039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5625.715961] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5628.248254] Lustre: server umount lustre-MDT0001 complete [ 5639.476637] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 5642.239942] Key type lgssc unregistered [ 5642.497867] LNet: 169544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5642.505970] LNetError: 169544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5642.530042] LNet: Removed LNI 192.168.204.113@tcp [ 5643.277822] Key type .llcrypt unregistered [ 5643.282584] Key type ._llcrypt unregistered [ 5666.012657] Key type ._llcrypt registered [ 5666.016494] Key type .llcrypt registered [ 5666.121048] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5667.112977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5667.231388] alg: No test for adler32 (adler32-zlib) [ 5668.239498] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5668.473776] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 5670.152307] Key type lgssc registered [ 5671.369190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5678.888912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5678.908568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5680.343571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5684.660632] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5690.853314] LustreError: 170957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5690.876310] LustreError: 170957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5693.290762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5693.661974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5698.538820] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5707.227806] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5713.011426] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5720.357762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5720.678616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5725.814209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5726.984288] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5736.151566] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5744.617941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5744.637430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5744.647128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5744.649032] Lustre: Skipped 1 previous similar message [ 5744.670130] Lustre: Skipped 1 previous similar message [ 5745.328812] Lustre: server umount lustre-OST0000 complete [ 5749.730284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5749.741848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5749.748424] Lustre: Skipped 1 previous similar message [ 5754.851882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5754.861034] Lustre: Skipped 1 previous similar message [ 5755.370875] Lustre: server umount lustre-MDT0000 complete [ 5759.029916] LustreError: 173748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447705 with bad export cookie 1015100829787065219 [ 5759.032792] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5759.042467] LustreError: 173748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5759.334242] Lustre: server umount lustre-MDT0001 complete [ 5769.157117] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 5772.354274] Key type lgssc unregistered [ 5772.628610] LNet: 174502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5772.646231] LNetError: 174502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5772.681239] LNet: Removed LNI 192.168.204.113@tcp [ 5773.743491] Key type .llcrypt unregistered [ 5773.751861] Key type ._llcrypt unregistered [ 5809.269455] Key type ._llcrypt registered [ 5809.271919] Key type .llcrypt registered [ 5809.395560] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5810.730438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5810.755752] alg: No test for adler32 (adler32-zlib) [ 5811.925145] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5812.253224] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 5813.976224] Key type lgssc registered [ 5815.231712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5823.932353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5823.953342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5825.594293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.410882] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5835.749756] LustreError: 176205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5835.775900] LustreError: 176205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5838.582329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.110698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5842.812452] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5852.318678] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5859.558308] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5867.198970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5867.592735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5870.717829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5875.106491] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5882.901355] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5891.553985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5891.561247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5891.573976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5893.107248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5893.111149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5898.214773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5898.232599] Lustre: Skipped 1 previous similar message [ 5903.342240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5903.355550] Lustre: Skipped 2 previous similar messages [ 5904.352188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5904.551634] Lustre: server umount lustre-OST0000 complete [ 5908.452773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5908.455089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5908.468589] Lustre: Skipped 1 previous similar message [ 5918.690688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5918.704398] Lustre: Skipped 4 previous similar messages [ 5922.272149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5922.414736] Lustre: server umount lustre-MDT0000 complete [ 5923.810201] LustreError: 178647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5923.839613] LustreError: 178647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5925.408175] LustreError: 178048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447872 with bad export cookie 5748549553136925405 [ 5925.412263] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5925.416241] LustreError: 178048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5925.663895] Lustre: server umount lustre-MDT0001 complete [ 5934.817959] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 5937.465391] Key type lgssc unregistered [ 5937.807063] LNet: 179744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5937.818598] LNetError: 179744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5938.859230] LNet: Removed LNI 192.168.204.113@tcp [ 5939.641334] Key type .llcrypt unregistered [ 5939.648456] Key type ._llcrypt unregistered [ 5959.432409] Key type ._llcrypt registered [ 5959.434697] Key type .llcrypt registered [ 5959.599562] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5960.574228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5960.606939] alg: No test for adler32 (adler32-zlib) [ 5961.762643] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5962.123827] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 5963.912263] Key type lgssc registered [ 5965.337332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5976.720907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5976.753065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5978.037531] Lustre: *** cfs_fail_loc=135, val=0*** [ 5978.121551] LustreError: 181248:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5978.134926] LustreError: 181248:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.113@tcp: cfg command failed: rc = -2 [ 5978.147868] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5978.147868] [ 5978.164259] LustreError: MGC192.168.204.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5978.176328] LustreError: 181210:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5978.182031] LustreError: 181210:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5978.189579] LustreError: 181210:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5978.204518] LustreError: 181210:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5978.310991] Lustre: server umount lustre-MDT0000 complete [ 5978.314032] LustreError: 181210:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5985.708595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5986.288481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5991.829451] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5996.515706] LustreError: 181621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5996.537889] LustreError: 181621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6001.652052] LustreError: 181620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6002.368041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6002.797523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6007.715976] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6016.765713] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6022.665821] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6024.406157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6024.406958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6024.418526] Lustre: Skipped 1 previous similar message [ 6024.428378] Lustre: Skipped 1 previous similar message [ 6027.490850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6027.500463] Lustre: Skipped 1 previous similar message [ 6030.826213] Lustre: server umount lustre-MDT0000 complete [ 6032.866120] LustreError: 181621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6032.882665] LustreError: 181621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6035.437607] LustreError: 181600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447982 with bad export cookie 486323294917993668 [ 6035.441454] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6035.448027] LustreError: 181600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6035.706594] Lustre: server umount lustre-MDT0001 complete [ 6047.627561] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6050.183955] Key type lgssc unregistered [ 6050.546826] LNet: 183944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6050.563661] LNetError: 183944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6050.589924] LNet: Removed LNI 192.168.204.113@tcp [ 6051.560316] Key type .llcrypt unregistered [ 6051.564961] Key type ._llcrypt unregistered [ 6072.709302] Key type ._llcrypt registered [ 6072.713684] Key type .llcrypt registered [ 6072.861702] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6073.153418] Key type .llcrypt unregistered [ 6073.159291] Key type ._llcrypt unregistered [ 6090.124331] Key type ._llcrypt registered [ 6090.126739] Key type .llcrypt registered [ 6090.220992] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6091.305948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6091.446887] alg: No test for adler32 (adler32-zlib) [ 6092.576321] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6092.873785] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6094.597371] Key type lgssc registered [ 6095.589733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6103.103392] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6103.122727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6118.690601] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6123.809896] LustreError: 186079:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.113@tcp: failed processing log, type 4: rc = -110 [ 6150.496293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6156.341502] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6165.323811] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6172.798741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6172.935876] LustreError: 185415:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000088faf57 x1875323690944768/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6173.238992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6173.292106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6178.291725] LustreError: 186104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6178.311724] LustreError: 186104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6178.805982] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6183.401224] LustreError: 186105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6185.447889] LustreError: 187117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6185.458371] LustreError: 187117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6188.391376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6188.883282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6193.211439] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6202.162815] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6207.769710] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6211.093099] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6229.473592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6229.482547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6229.503208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6229.996920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6230.017957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6230.062211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6232.784248] Lustre: server umount lustre-OST0000 complete [ 6240.227912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6240.234903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6240.251495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6240.275683] Lustre: Skipped 1 previous similar message [ 6240.296631] Lustre: Skipped 1 previous similar message [ 6244.769954] Lustre: server umount lustre-MDT0000 complete [ 6245.346680] LustreError: 187122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6245.373445] LustreError: 187122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6249.444064] LustreError: 187853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6249.460935] LustreError: 187853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6249.582207] LustreError: 186086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448196 with bad export cookie 6357094122153505825 [ 6249.589038] LustreError: 185419:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000855693e9 x1875323690977408/t0(0) o41->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 0 dl 1788448203 ref 1 fl Rpc:RESQU/200/0 rc -5/-107 job:'' uid:0 gid:0 projid:4294967295 [ 6249.593173] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6249.597406] LustreError: 186086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6250.040724] Lustre: server umount lustre-MDT0001 complete [ 6261.889579] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6264.988391] Key type lgssc unregistered [ 6265.332266] LNet: 190054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6265.350554] LNetError: 190054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6265.378612] LNet: Removed LNI 192.168.204.113@tcp [ 6266.153972] Key type .llcrypt unregistered [ 6266.157202] Key type ._llcrypt unregistered [ 6290.638979] Key type ._llcrypt registered [ 6290.644292] Key type .llcrypt registered [ 6290.774568] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6291.909038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6292.285252] alg: No test for adler32 (adler32-zlib) [ 6293.342147] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6293.710220] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6295.539065] Key type lgssc registered [ 6297.002493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6308.285820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6308.326742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6309.980793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6315.581048] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6320.100625] LustreError: 191466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6320.123536] LustreError: 191466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6323.788310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6324.189420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6328.628621] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6337.845336] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6344.885175] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6353.545606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6353.949736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6354.945806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6360.086610] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6368.465918] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6374.845793] Lustre: Failing over lustre-MDT0000 [ 6375.282240] Lustre: server umount lustre-MDT0000 complete [ 6375.395695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6375.404294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6375.411921] LustreError: 191467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6378.469370] LustreError: 191466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6378.472275] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6378.495886] LustreError: 191466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6378.519964] Lustre: Skipped 1 previous similar message [ 6381.206660] LustreError: 191463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6381.218426] LustreError: 191463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6386.324269] LustreError: 191461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6386.348081] LustreError: 191461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6394.851210] Lustre: 190780:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788448325/real 1788448325] req@00000000618f85d2 x1875323901734016/t0(0) o400->MGC192.168.204.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788448341 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6394.901105] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6394.920479] LustreError: 192475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6394.944036] LustreError: 192475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6395.195688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.092058] LustreError: 190776:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009359a2e6 x1875323901739136/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6405.483281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6405.572866] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6406.812161] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6410.744720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6410.795368] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6410.821575] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6410.866422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6415.733231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6423.487710] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6435.823956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6435.829973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.861338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6437.229724] Lustre: server umount lustre-OST0000 complete [ 6441.443330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6441.445536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6441.461962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6441.461971] Lustre: Skipped 2 previous similar messages [ 6441.486359] Lustre: Skipped 2 previous similar messages [ 6446.565199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6446.575701] Lustre: Skipped 2 previous similar messages [ 6447.689668] Lustre: server umount lustre-MDT0000 complete [ 6451.694265] LustreError: 191462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6451.715171] LustreError: 191462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6451.800540] LustreError: 191447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448398 with bad export cookie 15417748158734232447 [ 6451.812278] LustreError: 191447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6451.817867] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6452.153268] Lustre: server umount lustre-MDT0001 complete [ 6465.068809] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6468.648036] Key type lgssc unregistered [ 6469.053248] LNet: 196362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6469.061966] LNetError: 196362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6469.101123] LNet: Removed LNI 192.168.204.113@tcp [ 6470.162810] Key type .llcrypt unregistered [ 6470.168489] Key type ._llcrypt unregistered [ 6494.569826] Key type ._llcrypt registered [ 6494.572192] Key type .llcrypt registered [ 6494.686083] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6495.609742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6495.753810] alg: No test for adler32 (adler32-zlib) [ 6496.863675] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6497.094187] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6498.824218] Key type lgssc registered [ 6500.128433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6508.009842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6508.030269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6509.464068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6514.219410] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6519.780612] LustreError: 197776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6519.804202] LustreError: 197776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6521.971920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6522.307641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6527.048317] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6535.435420] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6541.033422] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6547.269990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6547.651691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6554.666133] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6554.820854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6563.696715] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6566.766084] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6573.982396] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6594.006207] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6606.307738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6606.328359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.349183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6608.356692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6608.366893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6608.377705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6608.409364] Lustre: Skipped 1 previous similar message [ 6611.104894] Lustre: server umount lustre-OST0000 complete [ 6618.595239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6618.621883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6618.634285] Lustre: Skipped 1 previous similar message [ 6621.729558] Lustre: server umount lustre-MDT0000 complete [ 6623.727475] LustreError: 197776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6623.762725] LustreError: 197776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6626.916956] LustreError: 198514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448573 with bad export cookie 2550628070525231353 [ 6626.917987] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6626.939952] LustreError: 198514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6627.715476] Lustre: server umount lustre-MDT0001 complete [ 6641.127904] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6644.561908] Key type lgssc unregistered [ 6644.835149] LNet: 201658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6644.844828] LNetError: 201658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6644.862436] LNet: Removed LNI 192.168.204.113@tcp [ 6645.985192] Key type .llcrypt unregistered [ 6645.987317] Key type ._llcrypt unregistered [ 6668.417901] Key type ._llcrypt registered [ 6668.421719] Key type .llcrypt registered [ 6668.527893] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6669.910188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6670.007990] alg: No test for adler32 (adler32-zlib) [ 6671.178582] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6671.486098] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6673.243137] Key type lgssc registered [ 6674.600293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6683.041675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6683.058683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6684.493330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6689.764529] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6694.887484] LustreError: 202928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6694.905878] LustreError: 202928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6700.011429] LustreError: 202929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6700.097066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6700.509501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6705.913801] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6715.032295] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6720.999644] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6728.588706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6729.008176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6730.051138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6735.202098] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6744.797875] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6752.041164] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6765.718220] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6772.785665] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6776.427541] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6786.543384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6786.553129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6786.555697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6786.573077] Lustre: Skipped 1 previous similar message [ 6791.483316] Lustre: server umount lustre-OST0000 complete [ 6796.248963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6796.266464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6796.267780] Lustre: Skipped 1 previous similar message [ 6796.287871] Lustre: Skipped 2 previous similar messages [ 6801.380078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6801.390333] Lustre: Skipped 1 previous similar message [ 6802.533615] Lustre: server umount lustre-MDT0000 complete [ 6806.499077] LustreError: 204079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6806.521443] LustreError: 204079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6806.609225] LustreError: 202909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448753 with bad export cookie 6638787351116619138 [ 6806.617832] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6806.626734] LustreError: 202909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6807.047819] Lustre: server umount lustre-MDT0001 complete [ 6819.826319] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6823.926363] Key type lgssc unregistered [ 6824.334322] LNet: 207760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6824.344621] LNetError: 207760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6824.376611] LNet: Removed LNI 192.168.204.113@tcp [ 6825.307218] Key type .llcrypt unregistered [ 6825.310574] Key type ._llcrypt unregistered [ 6850.281471] Key type ._llcrypt registered [ 6850.283969] Key type .llcrypt registered [ 6850.383618] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6851.553104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6851.716258] alg: No test for adler32 (adler32-zlib) [ 6852.753650] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6852.951756] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6854.672428] Key type lgssc registered [ 6855.961904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6865.427948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6865.439518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.994696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6871.681514] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6877.160464] LustreError: 209173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6877.175648] LustreError: 209173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6881.129219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6881.398808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6885.754887] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6893.111643] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6898.881361] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6905.751211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6906.291827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6911.424906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6913.749329] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6923.452243] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6942.177228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6942.186135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6942.193240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6942.206877] Lustre: Skipped 1 previous similar message [ 6942.234310] Lustre: Skipped 1 previous similar message [ 6946.821787] Lustre: server umount lustre-OST0000 complete [ 6952.418823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6952.427074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6952.439566] Lustre: Skipped 1 previous similar message [ 6952.457559] Lustre: Skipped 1 previous similar message [ 6957.538373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6957.547625] Lustre: Skipped 1 previous similar message [ 6962.659799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6962.669095] Lustre: Skipped 1 previous similar message [ 6966.240847] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6966.548542] Lustre: server umount lustre-MDT0000 complete [ 6967.786230] LustreError: 209173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6967.825460] LustreError: 209173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6971.696398] LustreError: 209908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448918 with bad export cookie 6843720247680526688 [ 6971.716140] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6971.731636] LustreError: 209908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6972.154974] Lustre: server umount lustre-MDT0001 complete [ 6984.186610] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6986.962051] Key type lgssc unregistered [ 6987.201152] LNet: 212864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6987.208603] LNetError: 212864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6987.232775] LNet: Removed LNI 192.168.204.113@tcp [ 6988.069566] Key type .llcrypt unregistered [ 6988.075276] Key type ._llcrypt unregistered [ 7010.033329] Key type ._llcrypt registered [ 7010.036151] Key type .llcrypt registered [ 7010.176756] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 7011.625446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7011.635792] alg: No test for adler32 (adler32-zlib) [ 7012.814278] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7013.083102] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 7014.744175] Key type lgssc registered [ 7015.948430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7024.524498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7024.584213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7026.091886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7030.908869] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7035.376202] LustreError: 214130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7035.408032] LustreError: 214130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7040.481711] LustreError: 214131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7041.313335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7041.662308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7046.361561] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7055.418445] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7061.536346] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7067.972760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7068.364366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7070.457882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7074.874337] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7083.870037] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7102.434624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7102.439939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7102.452834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7102.467265] Lustre: Skipped 1 previous similar message [ 7102.493599] Lustre: Skipped 1 previous similar message [ 7107.554367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7107.567932] Lustre: Skipped 1 previous similar message [ 7108.677282] Lustre: server umount lustre-OST0000 complete [ 7112.675623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7112.681089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7112.694145] Lustre: Skipped 1 previous similar message [ 7117.798659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7117.817531] Lustre: Skipped 2 previous similar messages [ 7118.663688] Lustre: server umount lustre-MDT0000 complete [ 7122.915890] LustreError: 214130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7122.934460] LustreError: 214130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7123.000802] LustreError: 214111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449069 with bad export cookie 12779483842628365381 [ 7123.009993] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7123.015582] LustreError: 214111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7123.533598] Lustre: server umount lustre-MDT0001 complete [ 7134.194590] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7137.675885] Key type lgssc unregistered [ 7138.135668] LNet: 217819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7138.142632] LNetError: 217819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7138.167751] LNet: Removed LNI 192.168.204.113@tcp [ 7139.086134] Key type .llcrypt unregistered [ 7139.089382] Key type ._llcrypt unregistered [ 7161.884264] Key type ._llcrypt registered [ 7161.886841] Key type .llcrypt registered [ 7162.079281] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 7163.309667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7163.422740] alg: No test for adler32 (adler32-zlib) [ 7164.475989] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7164.781690] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 7166.520195] Key type lgssc registered [ 7167.666925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7176.855931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7176.874197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7178.413359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7182.866859] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7188.962416] LustreError: 219086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7188.977388] LustreError: 219086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7191.475429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7191.930061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7196.751806] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7205.409346] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7211.251719] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7218.802808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7219.138430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7224.237979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7225.417733] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7234.403884] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7245.280471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7245.369120] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7245.673272] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7252.056767] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7260.594430] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7262.441449] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7274.159283] Lustre: Permanently deactivating lustre-OST0001 [ 7274.191314] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7281.231243] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7292.433226] Lustre: Permanently reactivating lustre-OST0001 [ 7292.445113] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7292.449781] Lustre: Skipped 2 previous similar messages [ 7299.153572] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7299.167222] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7299.179422] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7299.199440] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7299.235214] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7304.685310] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7304.693084] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7304.707988] Lustre: Skipped 1 previous similar message [ 7304.712409] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7304.719452] Lustre: Skipped 1 previous similar message [ 7305.701846] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7309.128820] Lustre: server umount lustre-OST0001 complete [ 7310.825106] LustreError: 220907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7310.850582] LustreError: 220907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7313.385127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7313.408373] LustreError: Skipped 1 previous similar message [ 7313.418946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7313.451062] Lustre: Skipped 1 previous similar message [ 7313.464898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7313.472738] Lustre: Skipped 1 previous similar message [ 7315.938470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7315.939289] LustreError: 222941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7315.942335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7315.946434] Lustre: Skipped 1 previous similar message [ 7316.013293] LustreError: 222941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7321.064556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7321.067261] LustreError: 220909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7321.076519] Lustre: Skipped 1 previous similar message [ 7321.103760] LustreError: 220909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7326.191211] LustreError: 220910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.236027] LustreError: 220910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7327.200103] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7327.413586] Lustre: server umount lustre-OST0000 complete [ 7335.404632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7335.419668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7335.438726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7335.442802] Lustre: Skipped 2 previous similar messages [ 7338.186787] Lustre: server umount lustre-MDT0000 complete [ 7341.543927] LustreError: 219086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7341.568812] LustreError: 219086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7343.570141] LustreError: 219067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449290 with bad export cookie 3033497643702215671 [ 7343.581024] LustreError: 219067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7343.581124] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7344.409421] Lustre: server umount lustre-MDT0001 complete [ 7357.662652] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7361.118832] Key type lgssc unregistered [ 7361.467398] LNet: 224189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7361.474304] LNetError: 224189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7361.491266] LNet: Removed LNI 192.168.204.113@tcp [ 7362.302544] Key type .llcrypt unregistered [ 7362.304851] Key type ._llcrypt unregistered [ 7405.808651] Key type ._llcrypt registered [ 7405.813932] Key type .llcrypt registered [ 7405.984676] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 7406.757928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7406.906538] alg: No test for adler32 (adler32-zlib) [ 7407.884408] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7408.181828] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 7409.856504] Key type lgssc registered [ 7411.024432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7419.160986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7419.190823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7420.657764] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7420.698918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7421.012689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7426.966797] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7437.214726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7437.328133] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7437.362797] Lustre: 226398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7437.581609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7442.497413] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7451.286405] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7458.777345] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7466.119305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7466.557211] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7466.780591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7472.522763] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7474.842876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7479.343639] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7483.257663] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7490.871166] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7500.889833] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7508.748540] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7525.419628] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7525.428827] Lustre: Skipped 1 previous similar message [ 7548.715835] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7548.725865] Lustre: Skipped 2 previous similar messages [ 7585.133803] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7585.142443] Lustre: Skipped 3 previous similar messages [ 7657.257813] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7657.263249] Lustre: Skipped 8 previous similar messages [ 7669.218311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7669.234324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7669.259512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7669.743831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7671.002438] Lustre: server umount lustre-OST0000 complete [ 7674.853779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7674.880292] Lustre: Skipped 1 previous similar message [ 7674.883568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7674.896417] Lustre: Skipped 2 previous similar messages [ 7679.974632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7679.992663] Lustre: Skipped 1 previous similar message [ 7681.097885] Lustre: server umount lustre-MDT0000 complete [ 7685.322871] LustreError: 229077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449632 with bad export cookie 5194096153301980509 [ 7685.325271] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7685.332935] LustreError: 229077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7685.745931] Lustre: server umount lustre-MDT0001 complete [ 7697.354391] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7701.149741] Key type lgssc unregistered [ 7701.476590] LNet: 231221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7701.490333] LNetError: 231221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7701.506301] LNet: Removed LNI 192.168.204.113@tcp [ 7702.470189] Key type .llcrypt unregistered [ 7702.476694] Key type ._llcrypt unregistered [ 7736.259415] Key type ._llcrypt registered [ 7736.261042] Key type .llcrypt registered [ 7736.479082] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 7738.582262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7738.856421] alg: No test for adler32 (adler32-zlib) [ 7739.932613] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7740.178384] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 7741.912200] Key type lgssc registered [ 7743.765864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7754.271521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7754.331675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7756.012161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7760.857699] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7765.487742] LustreError: 232636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7765.509444] LustreError: 232636:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7769.986359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7770.381089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7774.805768] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7782.380221] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7787.851837] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7794.564260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7794.911404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.313905] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7801.084305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7807.076623] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7821.795477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7821.800897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7821.824265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7821.832603] Lustre: Skipped 1 previous similar message [ 7822.296542] Lustre: server umount lustre-OST0000 complete [ 7826.913614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7826.929191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7826.930540] Lustre: Skipped 1 previous similar message [ 7826.936295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7826.956217] Lustre: Skipped 1 previous similar message [ 7832.039393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7832.047376] Lustre: Skipped 1 previous similar message [ 7837.153587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7837.160160] Lustre: Skipped 1 previous similar message [ 7841.248159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7841.252380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7841.284189] Lustre: Skipped 1 previous similar message [ 7841.579445] Lustre: server umount lustre-MDT0000 complete [ 7845.716934] LustreError: 235216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449792 with bad export cookie 18210839848778763382 [ 7845.736340] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7845.744396] LustreError: 235216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7846.184673] Lustre: server umount lustre-MDT0001 complete [ 7860.359477] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7864.611458] Key type lgssc unregistered [ 7864.946147] LNet: 236331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7864.965922] LNetError: 236331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7866.025070] LNet: Removed LNI 192.168.204.113@tcp [ 7867.257684] Key type .llcrypt unregistered [ 7867.264064] Key type ._llcrypt unregistered [ 7890.324831] Key type ._llcrypt registered [ 7890.328887] Key type .llcrypt registered [ 7890.434344] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 7891.662386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7891.677923] alg: No test for adler32 (adler32-zlib) [ 7892.821574] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7893.101981] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 7894.816300] Key type lgssc registered [ 7896.010938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8167.445357] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 8170.624722] Key type lgssc unregistered [ 8170.918717] LNet: 238362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8170.929442] LNetError: 238362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8170.953751] LNet: Removed LNI 192.168.204.113@tcp [ 8171.848454] Key type .llcrypt unregistered [ 8171.850616] Key type ._llcrypt unregistered [ 8200.765981] Key type ._llcrypt registered [ 8200.770910] Key type .llcrypt registered [ 8200.938979] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 8219.505546] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 8221.047490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8221.110069] alg: No test for adler32 (adler32-zlib) [ 8222.191117] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8222.468113] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 8224.192358] Key type lgssc registered [ 8225.381693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8260.020829] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7959 sec ========== 11:43:24 (1788450204) [ 8262.309786] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:43:27 (1788450207) === [ 8266.216937] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:43:30 (1788450210) === [ 8310.150801] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 8312.661432] Key type lgssc unregistered [ 8312.898175] LNet: 243621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8312.915827] LNetError: 243621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8312.935264] LNet: Removed LNI 192.168.204.113@tcp [ 8313.758523] Key type .llcrypt unregistered [ 8313.762486] Key type ._llcrypt unregistered