[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 592935318 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003359] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005024] kvm-guest: setup PV IPIs [ 0.009333] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011015] pid_max: default: 32768 minimum: 301 [ 0.012167] LSM: Security Framework initializing [ 0.013009] Yama: becoming mindful. [ 0.015061] SELinux: Initializing. [ 0.017068] *** VALIDATE selinux *** [ 0.026577] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031484] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032184] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036061] *** VALIDATE tmpfs *** [ 0.037500] *** VALIDATE proc *** [ 0.038295] *** VALIDATE cgroup *** [ 0.039010] *** VALIDATE cgroup2 *** [ 0.040303] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042038] Spectre V2 : User space: Vulnerable [ 0.043009] Speculative Store Bypass: Vulnerable [ 0.047102] debug: unmapping init [mem 0xffffffff98859000-0xffffffff98860fff] [ 0.050325] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.052000] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.053026] ... version: 2 [ 0.054008] ... bit width: 48 [ 0.055015] ... generic registers: 4 [ 0.056014] ... value mask: 0000ffffffffffff [ 0.057014] ... max period: 00007fffffffffff [ 0.058020] ... fixed-purpose events: 3 [ 0.059016] ... event mask: 000000070000000f [ 0.060419] rcu: Hierarchical SRCU implementation. [ 0.062826] smp: Bringing up secondary CPUs ... [ 0.065030] x86: Booting SMP configuration: [ 0.066020] .... node #0, CPUs: #1 #2 #3 [ 0.077115] smp: Brought up 1 node, 4 CPUs [ 0.079020] smpboot: Max logical packages: 1 [ 0.080025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149032] node 0 deferred pages initialised in 67ms [ 0.157231] devtmpfs: initialized [ 0.159213] x86/mm: Memory block size: 128MB [ 0.163455] gcov: version magic: 0x41383552 [ 0.167241] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.168106] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.169683] pinctrl core: initialized pinctrl subsystem [ 0.170300] [ 0.171012] ************************************************************* [ 0.172065] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173019] ** ** [ 0.174070] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175158] ** ** [ 0.176021] ** This means that this kernel is built to expose internal ** [ 0.177024] ** IOMMU data structures, which may compromise security on ** [ 0.178037] ** your system. ** [ 0.179024] ** ** [ 0.180023] ** If you see this message and you are not debugging the ** [ 0.181021] ** kernel, report this immediately to your vendor! ** [ 0.182024] ** ** [ 0.183023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184022] ************************************************************* [ 0.186402] NET: Registered protocol family 16 [ 0.188467] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189264] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190291] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.192050] cpuidle: using governor menu [ 0.196479] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.199882] PCI: Using configuration type 1 for base access [ 0.203252] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214105] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.217030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225089] cryptd: max_cpu_qlen set to 1000 [ 0.227301] ACPI: Added _OSI(Module Device) [ 0.228025] ACPI: Added _OSI(Processor Device) [ 0.229034] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230026] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236964] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242000] ACPI: Interpreter enabled [ 0.244159] ACPI: PM: (supports S0 S3 S4 S5) [ 0.248016] ACPI: Using IOAPIC for interrupt routing [ 0.252187] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.258517] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.277229] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.279054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.282025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.286089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.289430] acpiphp: Slot [2] registered [ 0.291156] acpiphp: Slot [5] registered [ 0.292126] acpiphp: Slot [6] registered [ 0.294115] acpiphp: Slot [7] registered [ 0.295118] acpiphp: Slot [8] registered [ 0.296135] acpiphp: Slot [9] registered [ 0.297096] acpiphp: Slot [10] registered [ 0.298119] acpiphp: Slot [3] registered [ 0.299207] acpiphp: Slot [4] registered [ 0.300090] acpiphp: Slot [11] registered [ 0.302074] acpiphp: Slot [12] registered [ 0.303000] acpiphp: Slot [13] registered [ 0.303000] acpiphp: Slot [14] registered [ 0.304076] acpiphp: Slot [15] registered [ 0.305093] acpiphp: Slot [16] registered [ 0.307111] acpiphp: Slot [17] registered [ 0.308112] acpiphp: Slot [18] registered [ 0.310075] acpiphp: Slot [19] registered [ 0.311099] acpiphp: Slot [20] registered [ 0.312114] acpiphp: Slot [21] registered [ 0.313069] acpiphp: Slot [22] registered [ 0.314090] acpiphp: Slot [23] registered [ 0.315074] acpiphp: Slot [24] registered [ 0.316076] acpiphp: Slot [25] registered [ 0.317085] acpiphp: Slot [26] registered [ 0.319132] acpiphp: Slot [27] registered [ 0.322135] acpiphp: Slot [28] registered [ 0.323102] acpiphp: Slot [29] registered [ 0.325188] acpiphp: Slot [30] registered [ 0.327117] acpiphp: Slot [31] registered [ 0.330075] PCI host bridge to bus 0000:00 [ 0.332033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.334031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.336087] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.348036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.352066] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.355376] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.358235] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.361656] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.372014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.378057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.387035] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.390028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.396026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.402522] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.407827] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.416164] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.424098] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 15625 usecs [ 0.437181] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.448022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.480026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.492026] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.505830] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.519027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.536028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.565023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.579547] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.592022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.606025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.625025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.640000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.652036] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.660018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.686019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.702527] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.719025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.732017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.764023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.779838] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.796023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.808022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.828020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.843120] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.850017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.866021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.902022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.934356] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.938493] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.941694] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.945248] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.947308] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.953183] iommu: Default domain type: Passthrough [ 0.956674] SCSI subsystem initialized [ 0.959204] ACPI: bus type USB registered [ 0.961393] usbcore: registered new interface driver usbfs [ 0.963119] usbcore: registered new interface driver hub [ 0.967132] usbcore: registered new device driver usb [ 0.970221] pps_core: LinuxPPS API ver. 1 registered [ 0.972013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.979071] PTP clock support registered [ 0.982066] EDAC MC: Ver: 3.0.0 [ 0.984185] PCI: Using ACPI for IRQ routing [ 0.986000] NetLabel: Initializing [ 0.986014] NetLabel: domain hash size = 128 [ 0.987015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.990165] NetLabel: unlabeled traffic allowed by default [ 0.993436] vgaarb: loaded [ 0.994439] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.996019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.002363] clocksource: Switched to clocksource kvm-clock [ 1.162464] VFS: Disk quotas dquot_6.6.0 [ 1.166694] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.171935] *** VALIDATE ramfs *** [ 1.176923] *** VALIDATE hugetlbfs *** [ 1.181469] pnp: PnP ACPI init [ 1.190754] pnp: PnP ACPI: found 6 devices [ 1.231825] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.238173] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.242984] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.248239] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.251182] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.254208] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.257654] NET: Registered protocol family 2 [ 1.260506] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.266448] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.270947] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.276995] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.280433] TCP: Hash tables configured (established 65536 bind 65536) [ 1.284417] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.287533] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.291641] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.300885] NET: Registered protocol family 1 [ 1.305565] RPC: Registered named UNIX socket transport module. [ 1.308361] RPC: Registered udp transport module. [ 1.310558] RPC: Registered tcp transport module. [ 1.313017] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.323016] NET: Registered protocol family 44 [ 1.328299] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.334220] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.337371] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.340175] PCI: CLS 0 bytes, default 64 [ 1.342298] Unpacking initramfs... [ 3.989520] debug: unmapping init [mem 0xffff99617cc54000-0xffff99617ffbffff] [ 3.999921] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.002762] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.008163] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.870519] Initialise system trusted keyrings [ 4.872351] Key type blacklist registered [ 4.877798] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.902122] zbud: loaded [ 4.911732] *** VALIDATE nfs *** [ 4.913226] *** VALIDATE nfs4 *** [ 4.919681] pstore: using deflate compression [ 4.930158] Platform Keyring initialized [ 5.188234] NET: Registered protocol family 38 [ 5.191404] Key type asymmetric registered [ 5.193596] Asymmetric key parser 'x509' registered [ 5.195892] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.199583] io scheduler mq-deadline registered [ 5.201331] io scheduler kyber registered [ 5.203453] io scheduler bfq registered [ 5.205571] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.209283] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.213145] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.216926] ACPI: Power Button [PWRF] [ 5.223516] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.236564] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.259210] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.271618] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.316656] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.365840] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.393660] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.398554] Non-volatile memory driver v1.3 [ 5.400324] Linux agpgart interface v0.103 [ 5.443302] virtio_blk virtio1: [vda] 151304 512-byte logical blocks (77.5 MB/73.9 MiB) [ 5.447102] vda: detected capacity change from 0 to 77467648 [ 5.482344] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.485843] vdb: detected capacity change from 0 to 1073741824 [ 5.513103] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.516239] vdc: detected capacity change from 0 to 2621440000 [ 5.547899] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.553930] vdd: detected capacity change from 0 to 2621440000 [ 5.578910] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.581990] vde: detected capacity change from 0 to 4294967296 [ 5.605578] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.608554] vdf: detected capacity change from 0 to 4294967296 [ 5.626512] libphy: Fixed MDIO Bus: probed [ 5.654684] usbcore: registered new interface driver usbserial_generic [ 5.657216] usbserial: USB Serial support registered for generic [ 5.661154] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.665821] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.667966] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.670836] mousedev: PS/2 mouse device common for all mice [ 5.675180] rtc_cmos 00:05: RTC can wake from S4 [ 5.681271] rtc_cmos 00:05: registered as rtc0 [ 5.683730] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.687046] intel_pstate: CPU model not supported [ 5.692213] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.692838] hid: raw HID events driver (C) Jiri Kosina [ 5.701689] usbcore: registered new interface driver usbhid [ 5.704521] usbhid: USB HID core driver [ 5.706345] drop_monitor: Initializing network drop monitor service [ 5.710455] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.714283] Initializing XFRM netlink socket [ 5.717649] NET: Registered protocol family 10 [ 5.718041] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.721415] Segment Routing with IPv6 [ 5.743944] NET: Registered protocol family 17 [ 5.747384] mpls_gso: MPLS GSO support [ 5.762552] RAS: Correctable Errors collector initialized. [ 5.768169] AVX version of gcm_enc/dec engaged. [ 5.772549] AES CTR mode by8 optimization enabled [ 5.904328] sched_clock: Marking stable (5904211353, 0)->(7311778800, -1407567447) [ 5.910718] registered taskstats version 1 [ 5.914701] Loading compiled-in X.509 certificates [ 5.919306] zswap: loaded using pool lzo/zbud [ 5.959032] Key type big_key registered [ 5.978192] Key type encrypted registered [ 5.981488] ima: No TPM chip found, activating TPM-bypass! [ 5.985628] ima: Allocated hash algorithm: sha1 [ 5.988958] ima: No architecture policies found [ 5.992189] evm: Initialising EVM extended attributes: [ 5.995568] evm: security.selinux [ 5.997922] evm: security.ima [ 5.999951] evm: security.capability [ 6.002629] evm: HMAC attrs: 0x1 [ 6.007542] rtc_cmos 00:05: setting system clock to 2026-09-18 05:14:02 UTC (1789708442) [ 6.016969] debug: unmapping init [mem 0xffffffff99803000-0xffffffff999fffff] [ 6.021817] debug: unmapping init [mem 0xffffffff98582000-0xffffffff98858fff] [ 6.035800] Write protecting the kernel read-only data: 28672k [ 6.040834] debug: unmapping init [mem 0xffffffff96c03000-0xffffffff96dfffff] [ 6.047568] debug: unmapping init [mem 0xffffffff97514000-0xffffffff975fffff] [ 6.109968] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.133882] systemd[1]: Detected virtualization kvm. [ 6.139294] systemd[1]: Detected architecture x86-64. [ 6.143710] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.184630] systemd[1]: No hostname configured. [ 6.187289] systemd[1]: Set hostname to . [ 6.191466] random: systemd: uninitialized urandom read (16 bytes read) [ 6.196977] systemd[1]: Initializing machine ID from random generator. [ 6.359993] random: ln: uninitialized urandom read (6 bytes read) [ 6.565953] random: systemd: uninitialized urandom read (16 bytes read) [ 6.572321] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 6.585373] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.596481] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.573976] device-mapper: uevent: version 1.0.3 [ 8.576586] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.133197] virtio_net virtio0 ens2: renamed from eth0 [ 10.203943] random: fast init done [ 10.556874] scsi host0: ata_piix [ 10.579297] scsi host1: ata_piix [ 10.595510] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.603781] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.523094] random: crng init done [ 15.530091] random: 7 urandom warning(s) missed due to ratelimiting [ 17.875119] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File S[ 19.426437] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) ystems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.676264] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.310727] SELinux: Disabled at runtime. [ 22.416801] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.432745] systemd[1]: Detected virtualization kvm. [ 22.439692] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.167711] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.179382] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.191658] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.202643] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.214824] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.239305] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.251642] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ 24.538671] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.305084] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.452230] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.569154] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.318045] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.501584] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 33.167747] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 33.818447] NFS: Registering the id_resolver key type [ 33.823996] Key type id_resolver registered [ 33.828179] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg446-server login: [ 64.575488] libcfs: loading out-of-tree module taints kernel. [ 64.598867] Key type ._llcrypt registered [ 64.600380] Key type .llcrypt registered [ 64.678162] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 74.429110] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 75.118832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.126366] alg: No test for adler32 (adler32-zlib) [ 76.216815] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 76.646532] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 78.288204] Key type lgssc registered [ 78.942212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 86.557093] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 104.395880] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 110.129412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 110.139332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 111.296057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 111.312897] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 111.361847] Lustre: lustre-MDT0000: new disk, initializing [ 111.423656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 111.434495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 113.185223] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 119.457057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 119.501120] Lustre: 6484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 119.518283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 119.521719] Lustre: Skipped 1 previous similar message [ 119.566615] Lustre: lustre-MDT0001: new disk, initializing [ 119.603608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 119.620762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 119.624529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 121.296441] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 123.837106] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 127.685552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 127.839799] Lustre: lustre-OST0000: new disk, initializing [ 127.841672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 127.845708] Lustre: 8421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 127.878938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 130.266426] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 133.114111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 133.119134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 133.155520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 136.747069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 136.811640] Lustre: lustre-OST0001: new disk, initializing [ 136.813500] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 136.816541] Lustre: 9492:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 136.843720] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 139.198283] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 141.833750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 141.838294] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 141.881530] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 146.075839] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 152.205808] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 154.910415] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing check_logdir /tmp/testlogs/ [ 157.170875] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing yml_node [ 159.198831] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 160.366394] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 161.482182] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 162.209555] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 18 01:16:37 EDT 2026 [ 169.470881] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 170.226661] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 182.753761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 182.754270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 182.759622] Lustre: Skipped 3 previous similar messages [ 182.767254] Lustre: Skipped 3 previous similar messages [ 187.874648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 187.878652] Lustre: Skipped 3 previous similar messages [ 188.580762] Lustre: server umount lustre-MDT0000 complete [ 190.138738] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789708626 with bad export cookie 2459866104831367056 [ 190.139866] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 190.144706] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 192.992966] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 192.994061] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 192.999314] Lustre: Skipped 1 previous similar message [ 196.000796] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 196.003697] Lustre: Skipped 2 previous similar messages [ 196.396551] Lustre: server umount lustre-MDT0001 complete [ 198.050346] Lustre: server umount lustre-OST0000 complete [ 199.722753] Lustre: server umount lustre-OST0001 complete [ 202.517591] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 206.419692] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 221.405086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 221.561224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 221.579452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 221.624395] Lustre: lustre-MDT0000: new disk, initializing [ 221.663072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.673174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 223.405483] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 228.516736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 228.572271] Lustre: 15031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 228.714149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 230.376967] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 235.215698] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 238.150441] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 244.276543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.781941] Lustre: lustre-OST0000: new disk, initializing [ 244.784911] Lustre: Skipped 1 previous similar message [ 244.787913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 244.793619] Lustre: Skipped 2 previous similar messages [ 244.798332] Lustre: 16999:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 244.942607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.957607] Lustre: Skipped 1 previous similar message [ 246.207870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 246.212978] Lustre: Skipped 1 previous similar message [ 246.215472] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 246.343028] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 248.337815] hrtimer: interrupt took 5800916 ns [ 250.901396] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 261.070663] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 265.808681] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 266.045787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 272.289717] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 272.574321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 276.969084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 276.970672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 276.974959] LustreError: Skipped 1 previous similar message [ 277.002831] Lustre: Skipped 1 previous similar message [ 277.018665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 277.026039] Lustre: Skipped 1 previous similar message [ 281.665946] Lustre: server umount lustre-OST0000 complete [ 287.201734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.204062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.210498] Lustre: Skipped 1 previous similar message [ 287.222886] Lustre: Skipped 1 previous similar message [ 292.346232] LustreError: 15953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 292.380822] LustreError: 15953:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 292.872331] Lustre: server umount lustre-MDT0000 complete [ 297.441556] LustreError: 15043:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.475075] LustreError: 15043:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 298.470575] LustreError: 15040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.503564] LustreError: 15040:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 299.395191] LustreError: 15023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789708735 with bad export cookie 2459866104831368589 [ 299.406062] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 299.416651] LustreError: 15023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 300.363690] Lustre: server umount lustre-MDT0001 complete [ 306.792753] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:18:59 (1789708739) [ 316.573612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 317.379529] LustreError: 19266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.554444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 322.535607] LustreError: 19267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.342111] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 331.756903] LustreError: 19266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.799878] LustreError: 19266:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 338.404681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 345.079444] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 356.331191] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 362.673211] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 370.044753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 370.507789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 370.516650] Lustre: Skipped 1 previous similar message [ 376.856397] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 384.621305] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 399.334625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 399.349334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.368767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 400.343314] Lustre: server umount lustre-OST0000 complete [ 405.476296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 407.527384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 407.538311] Lustre: Skipped 1 previous similar message [ 413.745279] Lustre: server umount lustre-MDT0000 complete [ 416.738707] LustreError: 19266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 416.758620] LustreError: 19266:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 417.347662] LustreError: 19246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789708853 with bad export cookie 2459866104831369478 [ 417.355371] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 417.360958] LustreError: 19246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 417.947150] Lustre: server umount lustre-MDT0001 complete [ 429.351595] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 432.134581] Key type lgssc unregistered [ 432.488378] LNet: 22820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.493792] LNetError: 22820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 432.525479] LNet: Removed LNI 192.168.204.146@tcp [ 433.485496] Key type .llcrypt unregistered [ 433.488363] Key type ._llcrypt unregistered [ 461.078677] Key type ._llcrypt registered [ 461.081406] Key type .llcrypt registered [ 461.229613] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 462.210188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 462.261167] alg: No test for adler32 (adler32-zlib) [ 463.406772] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 463.659318] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 465.320880] Key type lgssc registered [ 466.495132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 474.318940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 474.346576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.933782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 480.486711] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 486.373261] LustreError: 24235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.383656] LustreError: 24235:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 488.766598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.117780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 493.239946] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 501.023887] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 506.185975] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 512.231727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 512.745373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 518.428360] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 520.959223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 525.283789] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 541.679734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.691386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 541.691735] Lustre: Skipped 1 previous similar message [ 541.700424] Lustre: Skipped 1 previous similar message [ 546.171278] Lustre: server umount lustre-OST0000 complete [ 550.881726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 550.889556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.902754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 551.905900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 551.909911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 555.741990] Lustre: server umount lustre-MDT0000 complete [ 557.026290] LustreError: 24236:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.049530] LustreError: 24236:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 559.554622] LustreError: 24214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789708996 with bad export cookie 10028492990645422705 [ 559.565627] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.148554] Lustre: server umount lustre-MDT0001 complete [ 572.916711] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 576.753280] Key type lgssc unregistered [ 577.160305] LNet: 28208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 577.166963] LNetError: 28208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 577.180816] LNet: Removed LNI 192.168.204.146@tcp [ 578.360101] Key type .llcrypt unregistered [ 578.361319] Key type ._llcrypt unregistered [ 603.278777] Key type ._llcrypt registered [ 603.280744] Key type .llcrypt registered [ 603.473673] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 604.594706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 604.789289] alg: No test for adler32 (adler32-zlib) [ 605.769321] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 605.995807] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 607.720269] Key type lgssc registered [ 609.106260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 618.281138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 618.297801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.905797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 624.878108] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 630.259564] LustreError: 29621:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.278614] LustreError: 29621:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 633.619229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 634.151755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 638.937540] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 646.369763] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 652.668487] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 665.682417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 666.092199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 672.080655] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 679.624939] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 689.655157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 699.875401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 699.878259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 699.878799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 699.881703] Lustre: Skipped 1 previous similar message [ 699.920131] Lustre: Skipped 1 previous similar message [ 703.450377] Lustre: server umount lustre-OST0000 complete [ 710.116496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 710.119222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 710.131944] Lustre: Skipped 1 previous similar message [ 715.237261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 715.245367] Lustre: Skipped 2 previous similar messages [ 715.684624] Lustre: server umount lustre-MDT0000 complete [ 719.238654] LustreError: 29601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789709155 with bad export cookie 15971282526459083713 [ 719.250205] LustreError: 29601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 719.252169] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 720.231274] Lustre: server umount lustre-MDT0001 complete [ 737.122231] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 740.886916] Key type lgssc unregistered [ 741.306953] LNet: 33519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.322470] LNetError: 33519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.348739] LNet: Removed LNI 192.168.204.146@tcp [ 742.277084] Key type .llcrypt unregistered [ 742.279208] Key type ._llcrypt unregistered [ 763.300286] Key type ._llcrypt registered [ 763.301921] Key type .llcrypt registered [ 763.375134] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 763.899336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 763.956046] alg: No test for adler32 (adler32-zlib) [ 764.900969] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 765.089120] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 766.704192] Key type lgssc registered [ 767.466422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 773.173817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 773.192537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 774.579316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 777.774675] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 784.095210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 784.405991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 786.943836] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 791.139291] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 794.210730] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 798.358979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 798.643378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 799.670703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 802.441404] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 807.109224] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 814.052928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.053161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 814.066081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 814.077761] Lustre: Skipped 1 previous similar message [ 815.473350] Lustre: server umount lustre-OST0000 complete [ 819.170206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.171610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 819.181553] Lustre: Skipped 2 previous similar messages [ 823.543301] Lustre: server umount lustre-MDT0000 complete [ 824.290416] LustreError: 34926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 824.300827] LustreError: 34926:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 825.464879] LustreError: 34907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789709261 with bad export cookie 1117623987523852461 [ 825.468989] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 825.470920] LustreError: 34907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 825.705751] Lustre: server umount lustre-MDT0001 complete [ 830.857333] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 832.287321] Key type lgssc unregistered [ 832.449800] LNet: 38453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.452984] LNetError: 38453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.462518] LNet: Removed LNI 192.168.204.146@tcp [ 832.869182] Key type .llcrypt unregistered [ 832.870611] Key type ._llcrypt unregistered [ 845.574032] Key type ._llcrypt registered [ 845.575123] Key type .llcrypt registered [ 845.627416] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 846.043992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 846.095894] alg: No test for adler32 (adler32-zlib) [ 847.030642] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 847.164701] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 848.777895] Key type lgssc registered [ 849.350303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 853.730417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 853.743724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.042447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.299770] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 861.546974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.750183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 863.743233] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 867.128892] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 869.567894] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 872.568636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 872.803829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.642785] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 879.406254] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 879.978375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 892.387532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 892.392045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.419673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 892.424075] Lustre: Skipped 1 previous similar message [ 895.916250] Lustre: lustre-OST0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 897.507627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 897.513839] Lustre: Skipped 1 previous similar message [ 898.399196] Lustre: server umount lustre-OST0000 complete [ 912.625317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 912.631399] Lustre: Skipped 1 previous similar message [ 912.634744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 912.638820] Lustre: Skipped 1 previous similar message [ 917.985247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 917.990244] Lustre: Skipped 1 previous similar message [ 918.926658] Lustre: server umount lustre-MDT0000 complete [ 920.721812] LustreError: 39822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789709357 with bad export cookie 7284841008259604360 [ 920.729808] LustreError: 39822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 920.730147] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 934.880529] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 935.091373] Lustre: server umount lustre-MDT0001 complete [ 940.339810] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 941.946333] Key type lgssc unregistered [ 942.135249] LNet: 43573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.139279] LNetError: 43573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.155162] LNet: Removed LNI 192.168.204.146@tcp [ 942.694621] Key type .llcrypt unregistered [ 942.696588] Key type ._llcrypt unregistered [ 955.348597] Key type ._llcrypt registered [ 955.350649] Key type .llcrypt registered [ 955.406286] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 955.941187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 955.952339] alg: No test for adler32 (adler32-zlib) [ 956.915769] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 957.064879] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 958.664195] Key type lgssc registered [ 959.270238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 963.337833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 963.346678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 964.627363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 966.816160] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 971.130088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.349576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 973.322335] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 977.164924] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 980.032271] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 983.628258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 983.856521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 986.604395] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 989.916173] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 989.996110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1001.381472] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 1001.443497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.449096] Lustre: Skipped 2 previous similar messages [ 1003.523213] Lustre: server umount lustre-MDT0000 complete [ 1005.254567] LustreError: 44940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789709441 with bad export cookie 2409643178243701082 [ 1005.258717] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1005.260177] LustreError: 44940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1006.491314] LustreError: 44954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.502202] LustreError: 44954:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1006.507395] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 1006.509692] Lustre: Skipped 3 previous similar messages [ 1006.561421] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.594834] LustreError: 44955:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1010.595337] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1010.602839] LustreError: 44955:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1010.606239] Lustre: Skipped 1 previous similar message [ 1011.536544] Lustre: server umount lustre-MDT0001 complete [ 1022.627275] LustreError: 44940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.46@tcp arrived at 1789709459 with bad export cookie 2409643178243701964 [ 1022.634418] LustreError: 44940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1038.304245] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1038.453446] Lustre: server umount lustre-OST0000 complete [ 1046.679185] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1048.057891] Key type lgssc unregistered [ 1048.223961] LNet: 48815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.230774] LNetError: 48815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1048.242848] LNet: Removed LNI 192.168.204.146@tcp [ 1048.686909] Key type .llcrypt unregistered [ 1048.689063] Key type ._llcrypt unregistered [ 1060.946520] Key type ._llcrypt registered [ 1060.948394] Key type .llcrypt registered [ 1061.000060] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1061.443168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1061.455387] alg: No test for adler32 (adler32-zlib) [ 1062.354820] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 1062.471548] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1064.080190] Key type lgssc registered [ 1064.578793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1067.896692] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1067.905813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1083.424391] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1088.544303] LustreError: 50175:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 1115.168218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1118.265351] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1122.799333] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1287.161217] Lustre: server umount lustre-OST0000 complete [ 1293.823349] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1295.027992] Key type lgssc unregistered [ 1295.187932] LNet: 51849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.191802] LNetError: 51849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1295.201089] LNet: Removed LNI 192.168.204.146@tcp [ 1295.587770] Key type .llcrypt unregistered [ 1295.589255] Key type ._llcrypt unregistered [ 1306.379366] Key type ._llcrypt registered [ 1306.381184] Key type .llcrypt registered [ 1306.429398] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1306.873698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1306.925190] alg: No test for adler32 (adler32-zlib) [ 1307.814545] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 1307.929219] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1309.520284] Key type lgssc registered [ 1310.057646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1313.670453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1313.679464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.971341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1316.812693] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1320.818824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1321.016087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1322.819460] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1326.165617] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1328.921465] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1332.174898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1332.402960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.931131] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1338.222983] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1339.322473] Lustre: server umount lustre-OST0000 complete [ 1341.408939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1341.412299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.420117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1345.506060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1345.508266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.529570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.532536] Lustre: Skipped 2 previous similar messages [ 1351.649635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1351.652662] Lustre: Skipped 1 previous similar message [ 1355.232132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1355.328793] Lustre: server umount lustre-MDT0000 complete [ 1356.768979] LustreError: 53234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1356.775952] LustreError: 53234:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1357.072549] LustreError: 53215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789709793 with bad export cookie 14618763797852663586 [ 1357.076236] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1357.077345] LustreError: 53215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1357.262626] Lustre: server umount lustre-MDT0001 complete [ 1361.888754] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1363.119778] Key type lgssc unregistered [ 1363.271591] LNet: 56755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.274834] LNetError: 56755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.287170] LNet: Removed LNI 192.168.204.146@tcp [ 1363.695892] Key type .llcrypt unregistered [ 1363.697987] Key type ._llcrypt unregistered [ 1374.069627] Key type ._llcrypt registered [ 1374.070918] Key type .llcrypt registered [ 1374.117217] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1374.476430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1374.488143] alg: No test for adler32 (adler32-zlib) [ 1375.393489] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 1375.508199] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1377.120226] Key type lgssc registered [ 1377.635046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1381.173227] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1381.184956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1396.705535] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1401.824717] LustreError: 58116:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 1428.448229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1430.942257] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1434.082842] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1437.336054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1437.633223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1437.680812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1439.566175] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1442.724104] LustreError: 59152:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1442.729664] LustreError: 59152:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1443.653693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.844831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1445.762383] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1449.452761] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1452.230409] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1454.051189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1454.053179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1454.054970] Lustre: Skipped 1 previous similar message [ 1454.060989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1459.172776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1459.176147] Lustre: Skipped 2 previous similar messages [ 1459.452837] Lustre: server umount lustre-OST0000 complete [ 1479.651046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.651186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1479.651942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1479.656073] Lustre: Skipped 1 previous similar message [ 1483.745801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1483.748607] Lustre: Skipped 2 previous similar messages [ 1488.867247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1488.873455] Lustre: Skipped 1 previous similar message [ 1490.400131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1490.477701] Lustre: server umount lustre-MDT0000 complete [ 1492.150514] LustreError: 58123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789709928 with bad export cookie 9293450789776310831 [ 1492.157538] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1498.419047] Lustre: server umount lustre-MDT0001 complete [ 1502.984538] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1504.189261] Key type lgssc unregistered [ 1504.343405] LNet: 61852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.346788] LNetError: 61852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1504.355439] LNet: Removed LNI 192.168.204.146@tcp [ 1504.732893] Key type .llcrypt unregistered [ 1504.734614] Key type ._llcrypt unregistered [ 1514.425594] Key type ._llcrypt registered [ 1514.427303] Key type .llcrypt registered [ 1514.465572] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1514.762475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1514.768314] alg: No test for adler32 (adler32-zlib) [ 1515.631097] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 1515.726723] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1517.313919] Key type lgssc registered [ 1517.746123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1520.662093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1520.671574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1521.917341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1523.437057] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1526.830855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1527.017474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1528.561311] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1531.138554] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1532.977792] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1535.368086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1535.552318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.743603] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1540.371538] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1546.855083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1557.475708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1557.476892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1557.484081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1557.485980] LustreError: Skipped 1 previous similar message [ 1557.488280] Lustre: Skipped 1 previous similar message [ 1558.980355] Lustre: server umount lustre-OST0000 complete [ 1562.593607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.597608] Lustre: Skipped 1 previous similar message [ 1562.599830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1562.602719] Lustre: Skipped 1 previous similar message [ 1566.556934] Lustre: server umount lustre-MDT0000 complete [ 1567.713978] LustreError: 63962:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1567.724522] LustreError: 63962:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1567.862078] LustreError: 65733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789710004 with bad export cookie 5091905713202782970 [ 1567.865079] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1567.868849] LustreError: 65733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1568.015631] Lustre: server umount lustre-MDT0001 complete [ 1571.812892] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1572.992944] Key type lgssc unregistered [ 1573.144505] LNet: 66782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.147544] LNetError: 66782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.158526] LNet: Removed LNI 192.168.204.146@tcp [ 1573.511556] Key type .llcrypt unregistered [ 1573.513042] Key type ._llcrypt unregistered [ 1584.360432] Key type ._llcrypt registered [ 1584.362148] Key type .llcrypt registered [ 1584.411580] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1584.834188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1584.881532] alg: No test for adler32 (adler32-zlib) [ 1585.773166] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 1585.884120] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1587.480192] Key type lgssc registered [ 1587.922995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1591.305851] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 01:40:26 (1789710026) [ 1593.908142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1593.920846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1595.186113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1596.794510] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1600.244561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1600.429424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1602.110375] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1605.091448] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1607.394563] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1610.040254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1610.255106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1612.576393] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1615.234199] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1620.516511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1628.948175] Lustre: Failing over lustre-MDT0000 [ 1629.155276] Lustre: server umount lustre-MDT0000 complete [ 1629.667695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.669675] LustreError: 70297:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1629.672635] Lustre: Skipped 2 previous similar messages [ 1629.679766] LustreError: 70297:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1632.786293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.837607] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1632.895259] Lustre: *** cfs_fail_loc=135, val=0*** [ 1632.938199] LustreError: 71569:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1632.940648] LustreError: 71569:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.146@tcp: cfg command failed: rc = -2 [ 1632.943905] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1632.943905] [ 1632.948476] LustreError: MGC192.168.204.146@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1632.952978] LustreError: 71557:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1632.956365] LustreError: 71557:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 1632.959562] LustreError: 71557:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 1632.962034] LustreError: 71557:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1632.973993] Lustre: server umount lustre-MDT0000 complete [ 1632.975621] LustreError: 71557:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1633.186521] LustreError: 70297:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1633.193350] LustreError: 70297:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1634.787462] LustreError: 68462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.797209] LustreError: 68462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1635.348707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1635.393496] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1635.484390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1635.507046] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1636.882531] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1638.297930] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1640.938455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1640.949350] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1640.969401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1645.024938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1645.028147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.032916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1646.051163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1646.051339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1646.054655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.068841] Lustre: server umount lustre-OST0000 complete [ 1651.169887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1651.169987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1651.170987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1651.170992] Lustre: Skipped 1 previous similar message [ 1651.175906] Lustre: Skipped 1 previous similar message [ 1655.644255] Lustre: server umount lustre-MDT0000 complete [ 1656.288950] LustreError: 68462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.295798] LustreError: 68462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1656.992936] LustreError: 68441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789710093 with bad export cookie 2795758890587384351 [ 1656.996288] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1656.997853] LustreError: 68441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1657.147208] Lustre: server umount lustre-MDT0001 complete [ 1660.919051] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1662.038834] Key type lgssc unregistered [ 1662.172296] LNet: 73339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.175656] LNetError: 73339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.182436] LNet: Removed LNI 192.168.204.146@tcp [ 1662.510217] Key type .llcrypt unregistered [ 1662.511765] Key type ._llcrypt unregistered [ 1672.412994] Key type ._llcrypt registered [ 1672.414198] Key type .llcrypt registered [ 1672.465376] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1672.801066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1672.845040] alg: No test for adler32 (adler32-zlib) [ 1673.717821] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 1673.819988] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1675.408213] Key type lgssc registered [ 1675.843573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1678.825731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1678.835667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1680.078291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1681.749534] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1685.082112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.248953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1686.747298] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1689.413468] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1691.522016] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1693.941548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1694.142725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1696.358189] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1699.143363] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1705.447621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1711.797195] Lustre: Failing over lustre-MDT0000 [ 1711.970563] Lustre: server umount lustre-MDT0000 complete [ 1714.597943] LustreError: 76561:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.606049] LustreError: 76561:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1715.595863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.647356] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1715.682810] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1715.684197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 1715.687363] Lustre: Skipped 2 previous similar messages [ 1715.689372] Lustre: Skipped 1 previous similar message [ 1715.711121] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1715.772552] LustreError: 77831:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1715.776513] LustreError: 77831:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.146@tcp: cfg command failed: rc = -9 [ 1715.780496] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1715.780496] [ 1715.785729] LustreError: MGC192.168.204.146@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1715.791542] LustreError: 77820:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1715.794990] LustreError: 77820:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 1715.798571] LustreError: 77820:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 1715.801266] LustreError: 77820:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1715.815181] Lustre: server umount lustre-MDT0000 complete [ 1715.816577] LustreError: 77820:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1718.300258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.361758] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1718.467138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.493299] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1719.707448] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1719.883752] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1723.878595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1723.892903] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1723.912985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1725.920779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1725.924478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1725.929992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1725.932749] Lustre: Skipped 1 previous similar message [ 1728.994696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1728.995365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1728.997351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1731.131558] Lustre: server umount lustre-OST0000 complete [ 1734.113591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1734.114178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.121746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.123839] Lustre: Skipped 2 previous similar messages [ 1738.655597] Lustre: server umount lustre-MDT0000 complete [ 1739.232982] LustreError: 74725:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1739.238574] LustreError: 74725:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1740.081175] LustreError: 74707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789710176 with bad export cookie 13454214362751691785 [ 1740.083641] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1740.085210] LustreError: 74707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1740.252517] Lustre: server umount lustre-MDT0001 complete [ 1744.201657] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1745.380744] Key type lgssc unregistered [ 1745.512544] LNet: 79606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.515434] LNetError: 79606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.525345] LNet: Removed LNI 192.168.204.146@tcp [ 1745.866659] Key type .llcrypt unregistered [ 1745.867988] Key type ._llcrypt unregistered [ 1750.171575] Key type ._llcrypt registered [ 1750.173022] Key type .llcrypt registered [ 1750.213776] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1750.318446] Key type .llcrypt unregistered [ 1750.319810] Key type ._llcrypt unregistered [ 1759.830677] Key type ._llcrypt registered [ 1759.832011] Key type .llcrypt registered [ 1759.874868] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1760.226517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1760.272395] alg: No test for adler32 (adler32-zlib) [ 1761.155721] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 1761.255646] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1762.856194] Key type lgssc registered [ 1763.267228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1766.312054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1766.320790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1767.565579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1769.270884] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1772.931086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1773.130305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1774.786282] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1777.701579] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1779.881202] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1782.957159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1783.174743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.526369] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1788.341496] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1794.472804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1804.771479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1804.772590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.774338] LustreError: Skipped 1 previous similar message [ 1804.781571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1804.784788] Lustre: Skipped 1 previous similar message [ 1808.250102] Lustre: server umount lustre-OST0000 complete [ 1809.889315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.894133] Lustre: Skipped 1 previous similar message [ 1809.896178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1809.898300] Lustre: Skipped 1 previous similar message [ 1815.009971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1815.016296] Lustre: Skipped 1 previous similar message [ 1815.734233] Lustre: server umount lustre-MDT0000 complete [ 1817.208331] LustreError: 81455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789710253 with bad export cookie 11983366504538922832 [ 1817.210458] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1817.213056] LustreError: 81455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1817.382663] Lustre: server umount lustre-MDT0001 complete [ 1821.232172] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1822.396952] Key type lgssc unregistered [ 1822.538231] LNet: 85026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.541527] LNetError: 85026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.550416] LNet: Removed LNI 192.168.204.146@tcp [ 1822.929143] Key type .llcrypt unregistered [ 1822.930673] Key type ._llcrypt unregistered [ 1833.498428] Key type ._llcrypt registered [ 1833.499885] Key type .llcrypt registered [ 1833.550331] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1833.936861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1833.981294] alg: No test for adler32 (adler32-zlib) [ 1834.862227] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 1834.976445] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1836.568291] Key type lgssc registered [ 1837.089042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1840.356135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1840.366325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.577202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.187309] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1846.617135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.789633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1848.401757] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1851.239926] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1853.326116] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1855.778362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1855.943660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.071028] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1860.747534] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1868.284616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1877.473406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1877.476508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.482703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1878.498931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1878.499635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1878.506284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.619205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.622408] Lustre: Skipped 2 previous similar messages [ 1888.224131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1888.326539] Lustre: server umount lustre-OST0000 complete [ 1892.832927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1892.835383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.840131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1898.977918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1898.980417] Lustre: Skipped 3 previous similar messages [ 1904.096135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1904.164079] Lustre: server umount lustre-MDT0000 complete [ 1905.563250] LustreError: 86393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789710342 with bad export cookie 6597295330966185850 [ 1905.566085] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1905.567993] LustreError: 86393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1905.733645] Lustre: server umount lustre-MDT0001 complete [ 1909.738055] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1910.886242] Key type lgssc unregistered [ 1911.033679] LNet: 89964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.037264] LNetError: 89964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.046349] LNet: Removed LNI 192.168.204.146@tcp [ 1911.433836] Key type .llcrypt unregistered [ 1911.435612] Key type ._llcrypt unregistered [ 1921.251690] Key type ._llcrypt registered [ 1921.252993] Key type .llcrypt registered [ 1921.296555] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1921.737818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1921.746644] alg: No test for adler32 (adler32-zlib) [ 1922.613161] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 1922.722636] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1924.306755] Key type lgssc registered [ 1924.790123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1927.739919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1927.748299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.961727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.393392] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1933.669256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1933.829273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1935.265269] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1937.780433] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1939.930569] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1942.642642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1942.861470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1945.243571] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1948.115250] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1955.182756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1964.514726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1964.520053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.527984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1965.538500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1965.539287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.541707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1968.716540] Lustre: server umount lustre-OST0000 complete [ 1970.657325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1970.658097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1970.662151] Lustre: Skipped 1 previous similar message [ 1970.664106] Lustre: Skipped 1 previous similar message [ 1975.777202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1975.779885] Lustre: Skipped 2 previous similar messages [ 1976.346353] Lustre: server umount lustre-MDT0000 complete [ 1977.716555] LustreError: 91331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789710414 with bad export cookie 11830683509066567652 [ 1977.720474] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1977.720868] LustreError: 91331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1977.881076] Lustre: server umount lustre-MDT0001 complete [ 1982.059905] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1983.249821] Key type lgssc unregistered [ 1983.388445] LNet: 94901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1983.391069] LNetError: 94901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.400375] LNet: Removed LNI 192.168.204.146@tcp [ 1983.757995] Key type .llcrypt unregistered [ 1983.759927] Key type ._llcrypt unregistered [ 1993.808895] Key type ._llcrypt registered [ 1993.810277] Key type .llcrypt registered [ 1993.854596] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1994.275785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1994.283519] alg: No test for adler32 (adler32-zlib) [ 1995.156480] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 1995.258482] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1996.849889] Key type lgssc registered [ 1997.245179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2000.187542] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2000.200119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2015.712409] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2020.832238] LustreError: 96260:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 2047.456237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2049.569527] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2052.145661] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2054.148223] Lustre: server umount lustre-OST0000 complete [ 2056.991622] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 01:48:12 (1789710492) [ 2059.377732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2059.582073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.132721] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2064.372868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.515651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2066.011525] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2068.981300] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2071.253935] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2074.124456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2074.323799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.632821] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2079.610797] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2084.584092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2094.051452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2094.051955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.061531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2094.065255] Lustre: Skipped 1 previous similar message [ 2099.174135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2099.177849] Lustre: Skipped 1 previous similar message [ 2104.291184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2104.294160] Lustre: Skipped 1 previous similar message [ 2108.384143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2108.510772] Lustre: server umount lustre-OST0000 complete [ 2110.945348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2110.950026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.957944] Lustre: Skipped 1 previous similar message [ 2110.960944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2114.530635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2118.402497] Lustre: server umount lustre-MDT0000 complete [ 2119.649738] LustreError: 97896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.656756] LustreError: 97896:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2120.115700] LustreError: 100650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789710556 with bad export cookie 16104965828776643486 [ 2120.118833] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2120.123645] LustreError: 100650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2120.310215] Lustre: server umount lustre-MDT0001 complete [ 2124.786815] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2126.034993] Key type lgssc unregistered [ 2126.179612] LNet: 101606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.184115] LNetError: 101606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2126.193445] LNet: Removed LNI 192.168.204.146@tcp [ 2126.570572] Key type .llcrypt unregistered [ 2126.572539] Key type ._llcrypt unregistered [ 2144.095943] Key type ._llcrypt registered [ 2144.098264] Key type .llcrypt registered [ 2144.143446] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2144.526957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2144.580386] alg: No test for adler32 (adler32-zlib) [ 2145.455243] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 2145.571494] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2147.168207] Key type lgssc registered [ 2147.611843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2150.661107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2150.670302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.912450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2153.403437] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2156.514059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.690167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2158.173484] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2160.807619] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2162.793059] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2165.282616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2165.462371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.591809] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2170.134258] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2176.746464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2187.234677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.236229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2187.236389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2187.240861] Lustre: Skipped 1 previous similar message [ 2189.505151] Lustre: server umount lustre-OST0000 complete [ 2192.354098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.355875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2192.359037] Lustre: Skipped 1 previous similar message [ 2192.361082] Lustre: Skipped 1 previous similar message [ 2197.079845] Lustre: server umount lustre-MDT0000 complete [ 2197.474601] LustreError: 103760:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2197.480764] LustreError: 103760:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2198.385569] LustreError: 103744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789710634 with bad export cookie 8178543807494554186 [ 2198.387852] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2198.392434] LustreError: 103744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2198.570400] Lustre: server umount lustre-MDT0001 complete [ 2202.605463] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2203.815890] Key type lgssc unregistered [ 2203.964700] LNet: 107310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.969600] LNetError: 107310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.980624] LNet: Removed LNI 192.168.204.146@tcp [ 2204.332214] Key type .llcrypt unregistered [ 2204.333639] Key type ._llcrypt unregistered [ 2211.962738] Key type ._llcrypt registered [ 2211.963794] Key type .llcrypt registered [ 2212.002573] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2212.364180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2212.406070] alg: No test for adler32 (adler32-zlib) [ 2213.270764] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 2213.371857] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2214.960218] Key type lgssc registered [ 2215.339186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2218.005606] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2218.011894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2233.504342] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2238.624341] LustreError: 108525:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 2265.248760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2267.551121] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2270.238526] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2272.704350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.868987] LustreError: MGC192.168.204.146@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2272.873952] LustreError: 109549:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2272.876518] LustreError: 109549:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2272.880894] LustreError: 109549:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2272.957561] Lustre: server umount lustre-MDT0000 complete [ 2272.959679] LustreError: 109549:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2286.119775] Lustre: server umount lustre-OST0000 complete [ 2289.129349] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2292.160646] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2304.635953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2304.767462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2304.779462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2304.819595] Lustre: lustre-MDT0000: new disk, initializing [ 2304.846852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2304.854839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2306.346239] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2311.114471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2311.166600] Lustre: 112336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2311.192590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2311.194741] Lustre: Skipped 1 previous similar message [ 2311.232859] Lustre: lustre-MDT0001: new disk, initializing [ 2311.271418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2311.288368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2311.292161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2312.778351] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2316.871089] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2318.872698] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2321.320931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2321.460557] Lustre: lustre-OST0000: new disk, initializing [ 2321.462671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2321.465848] Lustre: 114303:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2321.495678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.541987] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2325.118042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2325.122288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2325.158619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2327.787618] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2329.441807] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2329.531024] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2331.108907] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2331.188695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2335.717242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2335.717653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2335.719811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2335.720564] LustreError: Skipped 1 previous similar message [ 2335.725365] Lustre: Skipped 1 previous similar message [ 2338.043133] Lustre: server umount lustre-OST0000 complete [ 2340.834536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.835568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2340.839257] Lustre: Skipped 1 previous similar message [ 2340.841555] Lustre: Skipped 1 previous similar message [ 2345.570076] Lustre: server umount lustre-MDT0000 complete [ 2346.854300] LustreError: 114301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789710783 with bad export cookie 2802003884418119669 [ 2346.857735] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2346.858278] LustreError: 114301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2347.011357] Lustre: server umount lustre-MDT0001 complete [ 2349.920752] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 01:53:05 (1789710785) [ 2351.169094] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2356.593647] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2359.779743] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2371.631220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.770218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2371.781711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2371.817719] Lustre: lustre-MDT0000: new disk, initializing [ 2371.839157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2371.845378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2373.301985] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2377.804438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.863528] Lustre: 119192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2377.896305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2377.898901] Lustre: Skipped 1 previous similar message [ 2377.940148] Lustre: lustre-MDT0001: new disk, initializing [ 2378.016706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2378.027848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2379.501118] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2383.507784] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2385.621482] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2388.150747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2388.301248] Lustre: lustre-OST0000: new disk, initializing [ 2388.303900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2388.307564] Lustre: 121159:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2388.344342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2388.346859] Lustre: Skipped 1 previous similar message [ 2390.486049] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2393.593378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2393.596782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2393.626915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2394.141650] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2395.539777] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2395.613601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2396.978990] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2397.047720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2398.691078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2398.691463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.692144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2398.694579] LustreError: Skipped 1 previous similar message [ 2398.699043] Lustre: Skipped 1 previous similar message [ 2399.791218] Lustre: server umount lustre-OST0000 complete [ 2403.809728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2403.810058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2403.810606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2403.810610] Lustre: Skipped 1 previous similar message [ 2403.820142] Lustre: Skipped 1 previous similar message [ 2407.198685] Lustre: server umount lustre-MDT0000 complete [ 2408.497220] LustreError: 119185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789710844 with bad export cookie 2802003884418120551 [ 2408.500640] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2408.501478] LustreError: 119185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2408.648530] Lustre: server umount lustre-MDT0001 complete [ 2411.422625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2411.613119] LustreError: 123372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.621092] LustreError: 123372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2411.640637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.025046] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2415.871718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2417.434095] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2419.902372] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2421.938506] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2424.336882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2426.327067] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2428.630807] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2441.805060] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2447.330644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2447.337078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2447.346834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2447.350123] Lustre: Skipped 2 previous similar messages [ 2449.327592] Lustre: server umount lustre-OST0000 complete [ 2451.940075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2451.949222] LustreError: Skipped 1 previous similar message [ 2457.569807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2457.575951] Lustre: Skipped 3 previous similar messages [ 2459.962631] Lustre: server umount lustre-MDT0000 complete [ 2462.691771] LustreError: 123373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2462.706812] LustreError: 123373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2463.351888] LustreError: 123353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789710899 with bad export cookie 2802003884418121433 [ 2463.368019] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2463.370023] LustreError: 123353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2463.746411] Lustre: server umount lustre-MDT0001 complete [ 2473.009175] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2476.007395] Key type lgssc unregistered [ 2476.372059] LNet: 127019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2476.380364] LNetError: 127019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2477.418322] LNet: Removed LNI 192.168.204.146@tcp [ 2478.671872] Key type .llcrypt unregistered [ 2478.674122] Key type ._llcrypt unregistered [ 2493.803906] Key type ._llcrypt registered [ 2493.808441] Key type .llcrypt registered [ 2493.880453] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2502.970507] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2503.569690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2503.594858] alg: No test for adler32 (adler32-zlib) [ 2504.616850] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 2504.789569] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2506.400403] Key type lgssc registered [ 2507.374838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2525.174167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2525.188381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2526.324091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2526.337261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2526.374387] Lustre: lustre-MDT0000: new disk, initializing [ 2526.406246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.416494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2528.260163] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2533.923136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.998451] Lustre: 130144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2534.039182] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2534.041445] Lustre: Skipped 1 previous similar message [ 2534.079899] Lustre: lustre-MDT0001: new disk, initializing [ 2534.141682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2534.171651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2534.177075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2536.214536] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2541.380969] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2543.998785] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2547.194655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2547.394951] Lustre: lustre-OST0000: new disk, initializing [ 2547.397509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2547.400925] Lustre: 132109:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2547.443360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2550.181065] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2554.893754] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2556.887060] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2557.440982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2557.445564] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2557.493535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2558.015783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2559.771534] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2559.864621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2562.531561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2562.532083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.534856] LustreError: Skipped 1 previous similar message [ 2562.540380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2562.542970] Lustre: Skipped 1 previous similar message [ 2566.885800] Lustre: server umount lustre-OST0000 complete [ 2570.208928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2570.212429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2570.217024] Lustre: Skipped 1 previous similar message [ 2570.220021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.770254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.770960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.956442] Lustre: server umount lustre-MDT0000 complete [ 2576.739571] LustreError: 130137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789711013 with bad export cookie 7772867585996604926 [ 2576.743910] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.746490] LustreError: 130137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2576.982599] Lustre: server umount lustre-MDT0001 complete [ 2580.665308] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 01:56:55 (1789711015) [ 2583.411725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.726215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.599563] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2589.155735] LustreError: 134527:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.167457] LustreError: 134527:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2589.512490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.699597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2591.537605] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2594.752125] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2597.211130] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2598.049036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2598.059536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2598.062375] Lustre: Skipped 2 previous similar messages [ 2604.207711] Lustre: server umount lustre-MDT0000 complete [ 2605.540143] LustreError: 135764:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.549367] LustreError: 135764:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2606.188504] LustreError: 136093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789711042 with bad export cookie 7772867585996605808 [ 2606.193279] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2606.195022] LustreError: 136093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2606.398258] Lustre: server umount lustre-MDT0001 complete [ 2610.369560] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 01:57:25 (1789711045) [ 2613.534838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2629.089601] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2634.208994] LustreError: 136939:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 2659.809217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.355720] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2665.169914] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2666.027264] Lustre: server umount lustre-OST0000 complete [ 2669.228602] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 01:58:24 (1789711104) [ 2671.665558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.891897] LustreError: 138382:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.406488] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2676.339982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.472061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2676.474022] Lustre: Skipped 1 previous similar message [ 2677.848344] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2680.373996] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2682.245736] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2684.520399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2686.772248] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2689.464835] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2702.307067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2702.307504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2702.307557] Lustre: Skipped 1 previous similar message [ 2702.309810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2702.319760] Lustre: Skipped 4 previous similar messages [ 2707.931970] Lustre: server umount lustre-MDT0000 complete [ 2709.319302] LustreError: 138363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789711145 with bad export cookie 7772867585996606459 [ 2709.322580] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2709.326926] LustreError: 138363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2709.477641] Lustre: server umount lustre-MDT0001 complete [ 2721.111160] Lustre: server umount lustre-OST0000 complete [ 2724.083959] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 01:59:19 (1789711159) [ 2726.196238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.377646] LustreError: 142054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.401413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.403532] Lustre: Skipped 1 previous similar message [ 2727.664811] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2730.410503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2731.809708] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2734.242437] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2736.115201] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2738.341124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2740.407457] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2742.909774] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2744.292414] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2751.846719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2752.502257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2753.936174] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2754.014025] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2755.040834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2755.043664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.048173] Lustre: Skipped 2 previous similar messages [ 2755.050558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2755.052833] Lustre: Skipped 3 previous similar messages [ 2760.818907] Lustre: server umount lustre-OST0000 complete [ 2768.216477] Lustre: server umount lustre-MDT0000 complete [ 2769.492249] LustreError: 142036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789711205 with bad export cookie 7772867585996607516 [ 2769.495628] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2769.496577] LustreError: 142036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2769.675722] Lustre: server umount lustre-MDT0001 complete [ 2772.560615] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:00:07 (1789711207) [ 2774.926323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2790.432663] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2795.552443] LustreError: 146267:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 2821.152353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.154154] Lustre: Skipped 2 previous similar messages [ 2823.006958] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2825.478483] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2827.706111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2827.875864] LustreError: 146292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.917147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2829.192532] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2831.868965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2833.329987] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2835.628326] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2837.386356] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2838.866305] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2838.946175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2840.319583] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2840.393407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2841.569461] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.571045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2841.573132] Lustre: Skipped 4 previous similar messages [ 2841.576518] Lustre: Skipped 8 previous similar messages [ 2855.392097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2855.442279] Lustre: server umount lustre-MDT0000 complete [ 2856.637281] LustreError: 146273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789711293 with bad export cookie 7772867585996608524 [ 2856.640117] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2856.640683] LustreError: 146273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2856.783131] Lustre: server umount lustre-MDT0001 complete [ 2868.217816] Lustre: server umount lustre-OST0000 complete [ 2870.946164] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 02:01:46 (1789711306) [ 2873.005573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2888.480496] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2893.600403] LustreError: 150471:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 2921.078031] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2923.519906] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2925.727407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.915562] LustreError: 150496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.958265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2927.250803] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2929.975986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.388837] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2933.820129] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2935.661245] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2937.921636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2937.991820] Lustre: lustre-OST0001: new disk, initializing [ 2937.993355] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2937.995559] Lustre: 153342:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2939.966078] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2943.701904] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2944.503496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2944.507714] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2944.537075] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2945.156667] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2945.226431] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2946.598539] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2946.682384] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2949.605298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.606793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2949.611032] Lustre: Skipped 2 previous similar messages [ 2949.612802] Lustre: Skipped 6 previous similar messages [ 2953.400209] Lustre: server umount lustre-OST0000 complete [ 2954.722489] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2954.723307] LustreError: 151543:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2954.724747] LustreError: Skipped 2 previous similar messages [ 2954.728675] LustreError: 151543:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2960.823382] Lustre: server umount lustre-OST0001 complete [ 2968.221731] Lustre: server umount lustre-MDT0000 complete [ 2969.392022] LustreError: 150479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789711405 with bad export cookie 7772867585996609476 [ 2969.394386] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2969.395281] LustreError: 150479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2969.540264] Lustre: server umount lustre-MDT0001 complete [ 2977.007549] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:03:32 (1789711412) [ 2977.582630] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2978.129393] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2978.669917] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:03:33 (1789711413) [ 2980.913817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2981.061269] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2981.073254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2981.153731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2981.156072] Lustre: Skipped 6 previous similar messages [ 2982.604628] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2985.699112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.742812] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2985.756690] Lustre: 156896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2987.308272] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2989.723043] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2991.532560] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2993.819931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2993.920409] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2995.877765] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2997.010519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2998.294350] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2999.689507] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2999.760785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3001.152631] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3001.221415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3002.339488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3002.342409] LustreError: Skipped 1 previous similar message [ 3003.961833] Lustre: server umount lustre-OST0000 complete [ 3017.859614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3019.984857] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3022.117362] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3022.122862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3022.122929] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3022.129836] Lustre: Skipped 1 previous similar message [ 3022.378867] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3023.940784] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3024.011914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3025.373378] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3025.449891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3027.233257] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff90ef09c1b000.ost_server_uuid 50 [ 3027.749079] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff90ef09c1b000.ost_server_uuid in FULL state after 0 sec [ 3027.819138] LustreError: 156901:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.46@tcp) returned error from blocking AST (req@000000004275a69d x1876647939859456 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000588c0869/0x6bdec5e31579e110 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.46@tcp remote: 0xc95e75565f6691ac expref: 5 pid: 156903 timeout: 3127 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3027.834032] LustreError: lustre-MDT0000: A client on nid 192.168.204.46@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3027.837764] LustreError: 156891:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.46@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000588c0869/0x6bdec5e31579e110 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.46@tcp remote: 0xc95e75565f6691ac expref: 6 pid: 156903 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3032.550304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3032.550381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.551270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3032.551275] Lustre: Skipped 9 previous similar messages [ 3032.556089] LustreError: Skipped 1 previous similar message [ 3032.561221] Lustre: Skipped 7 previous similar messages [ 3035.007731] Lustre: server umount lustre-OST0000 complete [ 3042.591262] Lustre: server umount lustre-MDT0000 complete [ 3042.785347] LustreError: 156907:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.791214] LustreError: 156907:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3043.925185] LustreError: 156888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789711480 with bad export cookie 7772867585996610659 [ 3043.928199] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3043.930041] LustreError: 156888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3050.129162] Lustre: server umount lustre-MDT0001 complete [ 3053.908775] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3054.888791] Key type lgssc unregistered [ 3055.007337] LNet: 162992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3055.009701] LNetError: 162992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3055.018293] LNet: Removed LNI 192.168.204.146@tcp [ 3055.324782] Key type .llcrypt unregistered [ 3055.325759] Key type ._llcrypt unregistered [ 3063.919866] Key type ._llcrypt registered [ 3063.921369] Key type .llcrypt registered [ 3063.960090] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3064.391582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3064.398809] alg: No test for adler32 (adler32-zlib) [ 3065.243577] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 3065.339164] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3066.920213] Key type lgssc registered [ 3067.249740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3069.647255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3069.652965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.830315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.141035] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3074.999025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.117828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3076.378463] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3078.695973] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3080.467078] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3082.583695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3082.723614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3084.473049] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3086.666280] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3096.038229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3099.056039] Lustre: Failing over lustre-MDT0000 [ 3099.281193] Lustre: server umount lustre-MDT0000 complete [ 3100.640814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3100.643061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.646833] LustreError: 164377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.650740] LustreError: 164377:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3101.153213] LustreError: 165115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3101.154186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.157311] LustreError: 165115:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3101.160623] Lustre: Skipped 1 previous similar message [ 3106.273130] LustreError: 164377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3106.278493] LustreError: 164377:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3110.175463] LustreError: 164357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.46@tcp arrived at 1789711546 with bad export cookie 16953121195337604756 [ 3110.180285] LustreError: 164357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3111.395103] LustreError: 165808:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3111.402541] LustreError: 165808:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3112.513372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.550776] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3112.636203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3112.658785] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3113.847671] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3114.557741] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3114.561317] Lustre: lustre-MDT0000: Denying connection for new client 9476b69f-dad5-42cd-a681-52da9f5a627c (at 192.168.204.46@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3118.053863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3123.172426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.173280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3123.176901] Lustre: Skipped 1 previous similar message [ 3127.415411] Lustre: server umount lustre-OST0000 complete [ 3128.584272] LustreError: 168122:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3128.586721] Lustre: 167476:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3128.590510] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3128.597053] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3128.600834] LustreError: 167476:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ff3e1f91 x1876648526908288/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3128.605625] LustreError: 167476:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3128.608262] LustreError: 167476:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3128.611191] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 3128.703720] Lustre: server umount lustre-MDT0000 complete [ 3129.873319] LustreError: 164359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789711566 with bad export cookie 16953121195337605113 [ 3129.876568] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3129.877231] LustreError: 164359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3129.883135] LustreError: 164377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.095226] Lustre: server umount lustre-MDT0001 complete [ 3139.726420] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3140.703899] Key type lgssc unregistered [ 3140.830398] LNet: 168873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3140.833422] LNetError: 168873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3140.843348] LNet: Removed LNI 192.168.204.146@tcp [ 3141.144616] Key type .llcrypt unregistered [ 3141.146228] Key type ._llcrypt unregistered [ 3149.913070] Key type ._llcrypt registered [ 3149.914044] Key type .llcrypt registered [ 3149.949170] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3150.256625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3150.269619] alg: No test for adler32 (adler32-zlib) [ 3151.114549] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 3151.193927] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3152.776164] Key type lgssc registered [ 3153.114076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3155.563936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3155.570688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3156.750205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3158.008879] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3160.679948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.806556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3162.109662] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3164.394352] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3166.145925] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3168.256212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3168.393718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3170.195109] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3172.574245] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3181.734033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3187.170830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3187.171986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3187.173521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3187.177198] Lustre: Skipped 1 previous similar message [ 3190.324700] Lustre: server umount lustre-OST0000 complete [ 3191.776691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3191.778671] LustreError: Skipped 1 previous similar message [ 3191.779983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.783885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3191.785597] Lustre: Skipped 1 previous similar message [ 3197.409170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3197.410978] Lustre: Skipped 3 previous similar messages [ 3197.591908] Lustre: server umount lustre-MDT0000 complete [ 3198.791439] LustreError: 172096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789711635 with bad export cookie 17506423878785856155 [ 3198.795811] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3198.795907] LustreError: 172096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3198.955225] Lustre: server umount lustre-MDT0001 complete [ 3202.269833] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3203.250416] Key type lgssc unregistered [ 3203.374339] LNet: 173806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3203.376256] LNetError: 173806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3203.383266] LNet: Removed LNI 192.168.204.146@tcp [ 3203.670918] Key type .llcrypt unregistered [ 3203.671834] Key type ._llcrypt unregistered [ 3214.396261] Key type ._llcrypt registered [ 3214.397650] Key type .llcrypt registered [ 3214.431455] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3214.787223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3214.794492] alg: No test for adler32 (adler32-zlib) [ 3215.626504] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 3215.705986] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3217.280149] Key type lgssc registered [ 3217.605504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3219.930266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3219.935756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3221.098554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3222.290726] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3224.841312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3224.953200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3226.155451] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3228.338467] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3229.981841] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3232.032734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3232.176535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3233.904026] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3236.028123] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3246.508105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3255.777538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3255.779739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.784210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3256.802655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3256.803130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3256.803132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3256.805399] Lustre: Skipped 1 previous similar message [ 3258.993809] Lustre: server umount lustre-OST0000 complete [ 3260.896715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3260.899034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3260.903202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3266.329389] Lustre: server umount lustre-MDT0000 complete [ 3267.041109] LustreError: 176206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3267.056511] LustreError: 176206:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3267.519446] LustreError: 177981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789711704 with bad export cookie 16731469443076223436 [ 3267.522335] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3267.522485] LustreError: 177981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3267.664035] Lustre: server umount lustre-MDT0001 complete [ 3270.949548] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3271.896213] Key type lgssc unregistered [ 3272.026382] LNet: 179031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3272.028563] LNetError: 179031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3272.036309] LNet: Removed LNI 192.168.204.146@tcp [ 3272.338725] Key type .llcrypt unregistered [ 3272.339585] Key type ._llcrypt unregistered [ 3280.357861] Key type ._llcrypt registered [ 3280.359049] Key type .llcrypt registered [ 3280.393734] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3280.743624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3280.781693] alg: No test for adler32 (adler32-zlib) [ 3281.634546] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 3281.720818] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3283.296134] Key type lgssc registered [ 3283.622887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3286.909953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3286.915951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3288.068890] Lustre: *** cfs_fail_loc=135, val=0*** [ 3288.105452] LustreError: 180526:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3288.107824] LustreError: 180526:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.146@tcp: cfg command failed: rc = -2 [ 3288.110898] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3288.110898] [ 3288.113950] LustreError: MGC192.168.204.146@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3288.118972] LustreError: 180488:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3288.121643] LustreError: 180488:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3288.124090] LustreError: 180488:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3288.126500] LustreError: 180488:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3288.193205] Lustre: server umount lustre-MDT0000 complete [ 3288.194463] LustreError: 180488:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3290.350865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3290.540318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3291.808323] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3294.521567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.648643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3295.893249] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3298.146947] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3299.871420] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3304.609624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.613344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3304.615311] Lustre: Skipped 1 previous similar message [ 3306.641307] Lustre: server umount lustre-MDT0000 complete [ 3307.844290] LustreError: 180876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789711744 with bad export cookie 918234367464773425 [ 3307.846845] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3307.848610] LustreError: 180876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3308.006329] Lustre: server umount lustre-MDT0001 complete [ 3311.438767] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3312.383293] Key type lgssc unregistered [ 3312.508350] LNet: 183212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3312.510944] LNetError: 183212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3312.521299] LNet: Removed LNI 192.168.204.146@tcp [ 3312.805401] Key type .llcrypt unregistered [ 3312.806367] Key type ._llcrypt unregistered [ 3319.297977] Key type ._llcrypt registered [ 3319.299239] Key type .llcrypt registered [ 3319.334647] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3319.435482] Key type .llcrypt unregistered [ 3319.437141] Key type ._llcrypt unregistered [ 3326.093310] Key type ._llcrypt registered [ 3326.094710] Key type .llcrypt registered [ 3326.135482] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3326.525028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3326.544781] alg: No test for adler32 (adler32-zlib) [ 3327.418567] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 3327.513730] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3329.096199] Key type lgssc registered [ 3329.466760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3332.083634] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3332.089911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3347.552369] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3352.672294] LustreError: 185311:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 3379.296191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.294634] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3383.816267] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3386.139942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3386.351879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3386.370639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3387.715634] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3390.605514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.736734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3392.085932] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3394.511402] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3396.388723] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3397.341681] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3404.263016] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3404.265059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3404.278586] Lustre: Skipped 1 previous similar message [ 3404.288923] Lustre: Skipped 1 previous similar message [ 3408.493929] Lustre: server umount lustre-OST0000 complete [ 3411.428231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3411.436408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.453568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3414.499104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3414.500267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3414.514014] Lustre: Skipped 1 previous similar message [ 3417.680544] Lustre: server umount lustre-MDT0000 complete [ 3419.622783] LustreError: 186351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3419.638193] LustreError: 186351:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3420.517164] LustreError: 185319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789711857 with bad export cookie 5505780050675166021 [ 3420.524530] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3420.528032] LustreError: 185319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3420.836674] Lustre: server umount lustre-MDT0001 complete [ 3440.085807] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3446.632075] Key type lgssc unregistered [ 3447.144361] LNet: 189239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3447.158623] LNetError: 189239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3447.245302] LNet: Removed LNI 192.168.204.146@tcp [ 3450.412239] Key type .llcrypt unregistered [ 3450.413649] Key type ._llcrypt unregistered [ 3509.955079] Key type ._llcrypt registered [ 3509.956704] Key type .llcrypt registered [ 3510.147341] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3511.694404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3511.789825] alg: No test for adler32 (adler32-zlib) [ 3513.013972] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 3513.292764] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3515.424599] Key type lgssc registered [ 3518.635726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3531.421934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3531.468371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3533.927700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3540.321574] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3543.523786] LustreError: 190654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3543.555099] LustreError: 190654:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3548.659210] LustreError: 190653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3549.553481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.160287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3555.612795] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3563.836764] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3569.254353] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3575.678968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3576.167974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3579.252464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3582.524440] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3589.523577] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3594.549556] Lustre: Failing over lustre-MDT0000 [ 3594.740255] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.765446] LustreError: 190649:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3594.893844] Lustre: server umount lustre-MDT0000 complete [ 3600.868172] LustreError: 190653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3600.880342] LustreError: 190653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 3605.987192] LustreError: 190648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3606.015635] LustreError: 190648:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3612.129912] Lustre: 189961:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789712032/real 1789712032] req@0000000074b49243 x1876648996659968/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789712048 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3612.149851] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3613.920135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3617.180547] LustreError: 191852:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3617.199342] LustreError: 191852:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 3622.639719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3622.735495] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3627.438446] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3627.788672] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3628.036623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3628.077493] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3628.136891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3632.640117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3640.746104] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3648.993587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3649.005192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3649.028102] Lustre: Skipped 2 previous similar messages [ 3649.042504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3649.505880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3649.818188] Lustre: server umount lustre-OST0000 complete [ 3656.680654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3656.692244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3656.710898] Lustre: Skipped 2 previous similar messages [ 3656.723457] Lustre: Skipped 2 previous similar messages [ 3660.309047] Lustre: server umount lustre-MDT0000 complete [ 3661.793976] LustreError: 191609:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3661.808031] LustreError: 191609:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 3664.842288] LustreError: 194239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789712101 with bad export cookie 3119825883035275360 [ 3664.843815] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3664.854947] LustreError: 194239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3665.314406] Lustre: server umount lustre-MDT0001 complete [ 3676.272086] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3679.002759] Key type lgssc unregistered [ 3679.281644] LNet: 195544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3679.294695] LNetError: 195544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3679.317296] LNet: Removed LNI 192.168.204.146@tcp [ 3680.115269] Key type .llcrypt unregistered [ 3680.117990] Key type ._llcrypt unregistered [ 3702.529305] Key type ._llcrypt registered [ 3702.533463] Key type .llcrypt registered [ 3702.632777] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3703.700260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3703.779882] alg: No test for adler32 (adler32-zlib) [ 3704.929549] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 3705.291232] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3706.984250] Key type lgssc registered [ 3708.098832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3716.231322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3716.253883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3717.752203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3722.433423] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3728.357737] LustreError: 196954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3728.387912] LustreError: 196954:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3732.065302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.584053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3738.469948] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3747.494412] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3753.679603] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3760.679803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3761.239995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3763.354393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3768.829055] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3777.978785] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3781.247942] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3789.444563] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3802.715522] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3817.953813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3817.954829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3817.969911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3817.993156] Lustre: Skipped 1 previous similar message [ 3817.999922] Lustre: Skipped 1 previous similar message [ 3823.086585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3823.094487] Lustre: Skipped 1 previous similar message [ 3823.494328] Lustre: server umount lustre-OST0000 complete [ 3829.743546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3829.751627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.776525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3833.315493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3833.316182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3834.593238] Lustre: server umount lustre-MDT0000 complete [ 3838.377594] LustreError: 196936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789712274 with bad export cookie 10862333536336816444 [ 3838.386897] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3838.396578] LustreError: 196936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3838.995135] Lustre: server umount lustre-MDT0001 complete [ 3850.267779] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3852.594911] Key type lgssc unregistered [ 3853.014675] LNet: 200828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3853.021127] LNetError: 200828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3853.048153] LNet: Removed LNI 192.168.204.146@tcp [ 3854.023485] Key type .llcrypt unregistered [ 3854.031898] Key type ._llcrypt unregistered [ 3878.309367] Key type ._llcrypt registered [ 3878.311881] Key type .llcrypt registered [ 3878.419657] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3879.783159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3879.858086] alg: No test for adler32 (adler32-zlib) [ 3880.980163] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 3881.258121] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3882.969575] Key type lgssc registered [ 3883.912055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3892.051747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3892.080532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.698580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3897.816658] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3903.977038] LustreError: 202092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.989411] LustreError: 202092:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3908.039535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3908.365592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3913.642458] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3922.568563] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3930.620282] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3941.281940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3941.705124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3948.882816] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3956.996399] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3964.283472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3974.396540] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3987.223449] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4000.061083] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4008.346725] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4019.693244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4019.719887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.790861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4020.710803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4020.714905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4020.719510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4024.769943] Lustre: server umount lustre-OST0000 complete [ 4030.949365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4030.962338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4030.971339] Lustre: Skipped 1 previous similar message [ 4030.989292] Lustre: Skipped 2 previous similar messages [ 4036.067160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4036.074406] Lustre: Skipped 1 previous similar message [ 4041.191486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4041.213621] Lustre: Skipped 1 previous similar message [ 4044.256181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4044.518472] Lustre: server umount lustre-MDT0000 complete [ 4046.305943] LustreError: 202092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4046.338449] LustreError: 202092:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4049.008288] LustreError: 202072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789712485 with bad export cookie 16555161097412733114 [ 4049.037912] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4049.046760] LustreError: 202072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4049.923311] Lustre: server umount lustre-MDT0001 complete [ 4064.173269] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4068.657335] Key type lgssc unregistered [ 4069.180627] LNet: 206899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4069.208066] LNetError: 206899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4069.244941] LNet: Removed LNI 192.168.204.146@tcp [ 4071.019987] Key type .llcrypt unregistered [ 4071.026510] Key type ._llcrypt unregistered [ 4104.478047] Key type ._llcrypt registered [ 4104.481201] Key type .llcrypt registered [ 4104.586920] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4106.800568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4106.902441] alg: No test for adler32 (adler32-zlib) [ 4108.025980] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 4108.256527] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4109.985094] Key type lgssc registered [ 4111.931407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4122.747853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4122.767025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4124.317292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4130.133269] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4133.857411] LustreError: 208320:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4133.906069] LustreError: 208320:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4138.980397] LustreError: 208319:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4140.759868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4141.151788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4146.653937] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4154.792171] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4161.350917] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4170.843931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4171.406615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.351920] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4192.865359] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4193.927728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4214.754550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4214.763447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4214.770775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4214.771565] Lustre: Skipped 1 previous similar message [ 4214.807906] Lustre: Skipped 1 previous similar message [ 4219.926687] Lustre: server umount lustre-OST0000 complete [ 4228.065865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4228.078127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.175601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4230.122263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4230.128126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4231.944670] Lustre: server umount lustre-MDT0000 complete [ 4235.235188] LustreError: 209048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4235.266703] LustreError: 209048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4239.724709] LustreError: 208298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789712676 with bad export cookie 17900202527211540432 [ 4239.733601] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4239.750073] LustreError: 208298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4241.123336] Lustre: server umount lustre-MDT0001 complete [ 4264.634579] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4269.198977] Key type lgssc unregistered [ 4269.588138] LNet: 212054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4269.597538] LNetError: 212054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4269.624352] LNet: Removed LNI 192.168.204.146@tcp [ 4270.673853] Key type .llcrypt unregistered [ 4270.680068] Key type ._llcrypt unregistered [ 4292.951950] Key type ._llcrypt registered [ 4292.955466] Key type .llcrypt registered [ 4293.070951] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4294.189265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4294.392160] alg: No test for adler32 (adler32-zlib) [ 4295.636466] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 4295.974415] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4297.656835] Key type lgssc registered [ 4298.866172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4307.655647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4307.689851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4309.401898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4314.347411] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4318.690893] LustreError: 213317:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4318.728463] LustreError: 213317:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4323.735948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4323.812420] LustreError: 213318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4324.293713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4329.034857] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4337.306533] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4344.156286] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4351.257845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4351.918697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.967771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4358.806526] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4367.156604] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4384.738688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4384.750647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.769514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4388.077514] Lustre: server umount lustre-OST0000 complete [ 4394.978407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4394.981169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4394.995849] Lustre: Skipped 2 previous similar messages [ 4395.006181] Lustre: Skipped 2 previous similar messages [ 4398.542305] Lustre: server umount lustre-MDT0000 complete [ 4400.102440] LustreError: 213313:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.126204] LustreError: 213313:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4402.935995] LustreError: 213298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789712839 with bad export cookie 4207617363209851342 [ 4402.993251] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4403.461923] Lustre: server umount lustre-MDT0001 complete [ 4416.026577] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4419.172663] Key type lgssc unregistered [ 4419.504256] LNet: 216998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4419.527134] LNetError: 216998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4420.582830] LNet: Removed LNI 192.168.204.146@tcp [ 4421.696127] Key type .llcrypt unregistered [ 4421.699129] Key type ._llcrypt unregistered [ 4442.840912] Key type ._llcrypt registered [ 4442.843922] Key type .llcrypt registered [ 4442.943072] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4443.785803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4443.893058] alg: No test for adler32 (adler32-zlib) [ 4445.145488] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 4445.463778] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4447.112566] Key type lgssc registered [ 4449.224811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4457.573382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4457.589508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4459.346384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4466.642920] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4468.714669] LustreError: 218266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4468.736375] LustreError: 218266:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4473.825216] LustreError: 218267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4478.951464] LustreError: 218266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4479.947948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.285976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4485.928984] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4494.620659] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4500.360376] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4509.549864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4510.409298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4518.060279] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4526.480737] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4528.842801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4542.846401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4543.062059] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4543.620806] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4553.306617] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4561.165752] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4562.620456] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4573.779495] Lustre: Permanently deactivating lustre-OST0001 [ 4573.816067] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4580.035825] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4591.043874] Lustre: Permanently reactivating lustre-OST0001 [ 4591.076118] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4591.081672] Lustre: Skipped 2 previous similar messages [ 4600.446604] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4600.472472] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4600.504341] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4600.544473] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4600.668435] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4605.922964] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4605.927834] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.942854] LustreError: Skipped 1 previous similar message [ 4605.948847] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4605.962303] Lustre: Skipped 2 previous similar messages [ 4606.953950] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4606.965427] Lustre: Skipped 2 previous similar messages [ 4610.586357] Lustre: server umount lustre-OST0001 complete [ 4612.102541] LustreError: 220089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4612.168274] LustreError: 220089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4617.187348] LustreError: 221336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4617.188098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4617.238167] LustreError: 221336:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4617.304315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4617.323961] Lustre: Skipped 1 previous similar message [ 4622.323930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4622.335546] Lustre: Skipped 1 previous similar message [ 4623.538233] Lustre: server umount lustre-OST0000 complete [ 4632.545388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.574337] Lustre: Skipped 1 previous similar message [ 4632.585770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4632.599035] Lustre: Skipped 1 previous similar message [ 4636.711467] Lustre: server umount lustre-MDT0000 complete [ 4637.666827] LustreError: 218261:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4637.687245] LustreError: 218261:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4641.193562] LustreError: 218247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789713077 with bad export cookie 14783540098439307375 [ 4641.197073] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4641.202281] LustreError: 218247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4642.056128] Lustre: server umount lustre-MDT0001 complete [ 4653.347964] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4656.424453] Key type lgssc unregistered [ 4656.808582] LNet: 223333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4656.827095] LNetError: 223333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4656.859372] LNet: Removed LNI 192.168.204.146@tcp [ 4658.045906] Key type .llcrypt unregistered [ 4658.049324] Key type ._llcrypt unregistered [ 4706.767850] Key type ._llcrypt registered [ 4706.774530] Key type .llcrypt registered [ 4706.894585] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4708.371539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4708.506680] alg: No test for adler32 (adler32-zlib) [ 4709.667254] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 4709.951412] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4711.656359] Key type lgssc registered [ 4713.220914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4721.686826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4721.712925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4723.062929] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4723.080405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4723.196916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4728.204834] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4737.074592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4737.187504] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4737.218790] Lustre: 225537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4737.568296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4742.445650] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4749.723865] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4755.556821] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4761.119801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4761.281509] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4761.487850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4767.521923] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4770.537887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 4774.936937] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4777.940681] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4784.544703] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4794.527584] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4802.484755] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4817.071631] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4817.092754] Lustre: Skipped 1 previous similar message [ 4833.754880] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4833.766851] Lustre: Skipped 1 previous similar message [ 4875.862627] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4875.866684] Lustre: Skipped 4 previous similar messages [ 4949.994691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4950.001814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4950.013798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4950.022177] Lustre: Skipped 1 previous similar message [ 4952.727911] Lustre: server umount lustre-OST0000 complete [ 4957.672297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4957.680123] LustreError: Skipped 1 previous similar message [ 4957.685883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4957.696738] Lustre: Skipped 1 previous similar message [ 4957.700694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4960.225415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4960.226139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.778480] Lustre: server umount lustre-MDT0000 complete [ 4965.829318] LustreError: 225528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789713402 with bad export cookie 7704218352240255952 [ 4965.839273] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4965.849568] LustreError: 225528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4966.252954] Lustre: server umount lustre-MDT0001 complete [ 4978.317884] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4982.045309] Key type lgssc unregistered [ 4982.353081] LNet: 230319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4982.364785] LNetError: 230319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4982.380854] LNet: Removed LNI 192.168.204.146@tcp [ 4983.470416] Key type .llcrypt unregistered [ 4983.472145] Key type ._llcrypt unregistered [ 5005.883645] Key type ._llcrypt registered [ 5005.888807] Key type .llcrypt registered [ 5006.038115] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5006.524476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5006.537464] alg: No test for adler32 (adler32-zlib) [ 5007.733560] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 5007.988422] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 5009.680212] Key type lgssc registered [ 5010.675432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5016.661471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5016.686372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5018.109022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5021.333167] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5027.841682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5028.184483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5031.868791] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5037.964578] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5042.120921] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5047.646675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5048.018229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5052.838166] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5058.328033] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5065.455759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 5081.060749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5081.061942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5081.068616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5081.073940] Lustre: Skipped 1 previous similar message [ 5081.082563] Lustre: Skipped 1 previous similar message [ 5085.649295] Lustre: server umount lustre-OST0000 complete [ 5089.765435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5089.770265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.784216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5091.298027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.298784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5091.310661] Lustre: Skipped 1 previous similar message [ 5094.392868] Lustre: server umount lustre-MDT0000 complete [ 5096.420159] LustreError: 231728:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5096.433328] LustreError: 231728:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5096.597101] LustreError: 234329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789713533 with bad export cookie 429367695191121589 [ 5096.601621] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5096.605693] LustreError: 234329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5096.826633] Lustre: server umount lustre-MDT0001 complete [ 5103.158026] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5104.754604] Key type lgssc unregistered [ 5104.951920] LNet: 235432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5104.958361] LNetError: 235432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5104.970503] LNet: Removed LNI 192.168.204.146@tcp [ 5105.538110] Key type .llcrypt unregistered [ 5105.540990] Key type ._llcrypt unregistered [ 5117.550600] Key type ._llcrypt registered [ 5117.553171] Key type .llcrypt registered [ 5117.610475] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5118.147308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5118.161283] alg: No test for adler32 (adler32-zlib) [ 5119.049616] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 5119.164702] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 5120.760235] Key type lgssc registered [ 5121.257361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5385.382684] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5386.414040] Key type lgssc unregistered [ 5386.548365] LNet: 237428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5386.552235] LNetError: 237428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5386.565461] LNet: Removed LNI 192.168.204.146@tcp [ 5386.862609] Key type .llcrypt unregistered [ 5386.863888] Key type ._llcrypt unregistered [ 5394.980117] Key type ._llcrypt registered [ 5394.981018] Key type .llcrypt registered [ 5395.016254] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5401.247869] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5401.640725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5401.648190] alg: No test for adler32 (adler32-zlib) [ 5402.488656] Lustre: Lustre: Build Version: 2.17.58_39_g8752880 [ 5402.570245] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 5404.152192] Key type lgssc registered [ 5404.539830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5414.506981] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5252 sec ========== 02:44:09 (1789713849) [ 5415.081543] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:44:10 (1789713850) === [ 5416.156133] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:44:11 (1789713851) === [ 5429.239485] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5430.297638] Key type lgssc unregistered [ 5430.417281] LNet: 242627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5430.419719] LNetError: 242627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5430.426218] LNet: Removed LNI 192.168.204.146@tcp [ 5430.738363] Key type .llcrypt unregistered [ 5430.739552] Key type ._llcrypt unregistered