[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 504164720 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002318] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007873] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.011011] LSM: Security Framework initializing [ 0.012071] Yama: becoming mindful. [ 0.013042] SELinux: Initializing. [ 0.014065] *** VALIDATE selinux *** [ 0.022602] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026761] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029124] *** VALIDATE tmpfs *** [ 0.030447] *** VALIDATE proc *** [ 0.031252] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.033263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036032] Spectre V2 : User space: Vulnerable [ 0.037009] Speculative Store Bypass: Vulnerable [ 0.039923] debug: unmapping init [mem 0xffffffffba459000-0xffffffffba460fff] [ 0.041170] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042699] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043026] ... version: 2 [ 0.044016] ... bit width: 48 [ 0.045013] ... generic registers: 4 [ 0.046014] ... value mask: 0000ffffffffffff [ 0.047013] ... max period: 00007fffffffffff [ 0.048011] ... fixed-purpose events: 3 [ 0.049011] ... event mask: 000000070000000f [ 0.050298] rcu: Hierarchical SRCU implementation. [ 0.052561] smp: Bringing up secondary CPUs ... [ 0.053616] x86: Booting SMP configuration: [ 0.054031] .... node #0, CPUs: #1 #2 #3 [ 0.059124] smp: Brought up 1 node, 4 CPUs [ 0.061018] smpboot: Max logical packages: 1 [ 0.062015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132910] node 0 deferred pages initialised in 67ms [ 0.136144] devtmpfs: initialized [ 0.137240] x86/mm: Memory block size: 128MB [ 0.140012] gcov: version magic: 0x41383552 [ 0.142270] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144277] pinctrl core: initialized pinctrl subsystem [ 0.145131] [ 0.145447] ************************************************************* [ 0.146013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.147013] ** ** [ 0.148015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.149012] ** ** [ 0.150015] ** This means that this kernel is built to expose internal ** [ 0.151014] ** IOMMU data structures, which may compromise security on ** [ 0.152014] ** your system. ** [ 0.153014] ** ** [ 0.154013] ** If you see this message and you are not debugging the ** [ 0.155015] ** kernel, report this immediately to your vendor! ** [ 0.156015] ** ** [ 0.157014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158015] ************************************************************* [ 0.159743] NET: Registered protocol family 16 [ 0.160529] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.161070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.162072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.163616] cpuidle: using governor menu [ 0.164836] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.167431] PCI: Using configuration type 1 for base access [ 0.169144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.180171] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.183023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187042] cryptd: max_cpu_qlen set to 1000 [ 0.191120] ACPI: Added _OSI(Module Device) [ 0.193014] ACPI: Added _OSI(Processor Device) [ 0.194014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.201200] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.207348] ACPI: Interpreter enabled [ 0.209055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210012] ACPI: Using IOAPIC for interrupt routing [ 0.212102] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.215540] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.226572] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.228047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.231022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.235119] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.240339] acpiphp: Slot [2] registered [ 0.242120] acpiphp: Slot [5] registered [ 0.243111] acpiphp: Slot [6] registered [ 0.245151] acpiphp: Slot [7] registered [ 0.247127] acpiphp: Slot [8] registered [ 0.248095] acpiphp: Slot [9] registered [ 0.251129] acpiphp: Slot [10] registered [ 0.253139] acpiphp: Slot [3] registered [ 0.255104] acpiphp: Slot [4] registered [ 0.256136] acpiphp: Slot [11] registered [ 0.258156] acpiphp: Slot [12] registered [ 0.260098] acpiphp: Slot [13] registered [ 0.261095] acpiphp: Slot [14] registered [ 0.263121] acpiphp: Slot [15] registered [ 0.264106] acpiphp: Slot [16] registered [ 0.266144] acpiphp: Slot [17] registered [ 0.268108] acpiphp: Slot [18] registered [ 0.270149] acpiphp: Slot [19] registered [ 0.272116] acpiphp: Slot [20] registered [ 0.273095] acpiphp: Slot [21] registered [ 0.275102] acpiphp: Slot [22] registered [ 0.276109] acpiphp: Slot [23] registered [ 0.278115] acpiphp: Slot [24] registered [ 0.280272] acpiphp: Slot [25] registered [ 0.282126] acpiphp: Slot [26] registered [ 0.283107] acpiphp: Slot [27] registered [ 0.285097] acpiphp: Slot [28] registered [ 0.286086] acpiphp: Slot [29] registered [ 0.288101] acpiphp: Slot [30] registered [ 0.289124] acpiphp: Slot [31] registered [ 0.290060] PCI host bridge to bus 0000:00 [ 0.291018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.294025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.296021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.299027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.301037] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.305030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.307204] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.311067] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.314412] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.324874] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.330630] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.332029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.335017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.339021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.341631] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.344675] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.347070] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.348944] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.354016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.368945] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.373013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.379145] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.386016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.393019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.409016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.418710] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.426017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.433018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.453017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.463479] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.474018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.483016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.506017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.517468] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.523016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.530014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.551014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.562157] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.569015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.575015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.592017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.603000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.610016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.618016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.636017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.647243] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.649373] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.651374] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.654490] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.657229] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.661115] iommu: Default domain type: Passthrough [ 0.663436] SCSI subsystem initialized [ 0.665112] ACPI: bus type USB registered [ 0.666087] usbcore: registered new interface driver usbfs [ 0.668089] usbcore: registered new interface driver hub [ 0.670082] usbcore: registered new device driver usb [ 0.672185] pps_core: LinuxPPS API ver. 1 registered [ 0.673011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.676077] PTP clock support registered [ 0.679086] EDAC MC: Ver: 3.0.0 [ 0.680619] PCI: Using ACPI for IRQ routing [ 0.682881] NetLabel: Initializing [ 0.685014] NetLabel: domain hash size = 128 [ 0.686010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.688102] NetLabel: unlabeled traffic allowed by default [ 0.690423] vgaarb: loaded [ 0.692280] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.693014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.700811] clocksource: Switched to clocksource kvm-clock [ 0.810448] VFS: Disk quotas dquot_6.6.0 [ 0.812395] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.815234] *** VALIDATE ramfs *** [ 0.816749] *** VALIDATE hugetlbfs *** [ 0.818924] pnp: PnP ACPI init [ 0.821253] pnp: PnP ACPI: found 6 devices [ 0.838797] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.842423] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.845180] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.847950] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.850758] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.853336] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.856178] NET: Registered protocol family 2 [ 0.858580] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.864028] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.870737] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.875681] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.879193] TCP: Hash tables configured (established 65536 bind 65536) [ 0.882067] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.884837] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.887594] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.890420] NET: Registered protocol family 1 [ 0.893712] RPC: Registered named UNIX socket transport module. [ 0.895436] RPC: Registered udp transport module. [ 0.896964] RPC: Registered tcp transport module. [ 0.898215] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.900358] NET: Registered protocol family 44 [ 0.901731] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.903403] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.905033] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.906764] PCI: CLS 0 bytes, default 64 [ 0.907992] Unpacking initramfs... [ 2.279195] debug: unmapping init [mem 0xffff9eeabcc54000-0xffff9eeabffbffff] [ 2.283488] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.285861] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.288563] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.783975] Initialise system trusted keyrings [ 2.785483] Key type blacklist registered [ 2.787150] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.795733] zbud: loaded [ 2.798462] *** VALIDATE nfs *** [ 2.799549] *** VALIDATE nfs4 *** [ 2.800891] pstore: using deflate compression [ 2.804261] Platform Keyring initialized [ 2.904979] NET: Registered protocol family 38 [ 2.906880] Key type asymmetric registered [ 2.908625] Asymmetric key parser 'x509' registered [ 2.910914] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.914636] io scheduler mq-deadline registered [ 2.916392] io scheduler kyber registered [ 2.918428] io scheduler bfq registered [ 2.920514] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.923790] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.926925] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.930055] ACPI: Power Button [PWRF] [ 2.935074] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.942318] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.954840] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.960984] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.975748] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.006658] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.034561] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.039388] Non-volatile memory driver v1.3 [ 3.041537] Linux agpgart interface v0.103 [ 3.073875] virtio_blk virtio1: [vda] 149944 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.077430] vda: detected capacity change from 0 to 76771328 [ 3.093920] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.096504] vdb: detected capacity change from 0 to 1073741824 [ 3.113882] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.116637] vdc: detected capacity change from 0 to 2621440000 [ 3.129044] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.132021] vdd: detected capacity change from 0 to 2621440000 [ 3.145681] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.148486] vde: detected capacity change from 0 to 4294967296 [ 3.161725] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.164038] vdf: detected capacity change from 0 to 4294967296 [ 3.173701] libphy: Fixed MDIO Bus: probed [ 3.178476] usbcore: registered new interface driver usbserial_generic [ 3.181164] usbserial: USB Serial support registered for generic [ 3.183268] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.187191] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.189059] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.191353] mousedev: PS/2 mouse device common for all mice [ 3.193996] rtc_cmos 00:05: RTC can wake from S4 [ 3.196680] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.200645] rtc_cmos 00:05: registered as rtc0 [ 3.203474] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.204661] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.206285] intel_pstate: CPU model not supported [ 3.212426] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.216042] hid: raw HID events driver (C) Jiri Kosina [ 3.218180] usbcore: registered new interface driver usbhid [ 3.219900] usbhid: USB HID core driver [ 3.221161] drop_monitor: Initializing network drop monitor service [ 3.223037] Initializing XFRM netlink socket [ 3.225075] NET: Registered protocol family 10 [ 3.228044] Segment Routing with IPv6 [ 3.229426] NET: Registered protocol family 17 [ 3.231290] mpls_gso: MPLS GSO support [ 3.237532] RAS: Correctable Errors collector initialized. [ 3.239478] AVX version of gcm_enc/dec engaged. [ 3.241166] AES CTR mode by8 optimization enabled [ 3.309145] sched_clock: Marking stable (3309083367, 0)->(4273063774, -963980407) [ 3.312362] registered taskstats version 1 [ 3.314466] Loading compiled-in X.509 certificates [ 3.316150] zswap: loaded using pool lzo/zbud [ 3.337825] Key type big_key registered [ 3.348145] Key type encrypted registered [ 3.350151] ima: No TPM chip found, activating TPM-bypass! [ 3.352120] ima: Allocated hash algorithm: sha1 [ 3.353436] ima: No architecture policies found [ 3.354876] evm: Initialising EVM extended attributes: [ 3.356891] evm: security.selinux [ 3.358042] evm: security.ima [ 3.358708] evm: security.capability [ 3.359615] evm: HMAC attrs: 0x1 [ 3.361733] rtc_cmos 00:05: setting system clock to 2026-09-13 12:23:50 UTC (1789302230) [ 3.368452] debug: unmapping init [mem 0xffffffffbb403000-0xffffffffbb5fffff] [ 3.371457] debug: unmapping init [mem 0xffffffffba182000-0xffffffffba458fff] [ 3.382104] Write protecting the kernel read-only data: 28672k [ 3.385191] debug: unmapping init [mem 0xffffffffb8803000-0xffffffffb89fffff] [ 3.388251] debug: unmapping init [mem 0xffffffffb9114000-0xffffffffb91fffff] [ 3.419654] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.427708] systemd[1]: Detected virtualization kvm. [ 3.429706] systemd[1]: Detected architecture x86-64. [ 3.431465] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.458827] systemd[1]: No hostname configured. [ 3.460077] systemd[1]: Set hostname to . [ 3.461717] random: systemd: uninitialized urandom read (16 bytes read) [ 3.463751] systemd[1]: Initializing machine ID from random generator. [ 3.508402] random: ln: uninitialized urandom read (6 bytes read) [ 3.583732] random: systemd: uninitialized urandom read (16 bytes read) [ 3.585877] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.591326] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.600618] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.191218] device-mapper: uevent: version 1.0.3 [ 4.192693] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.992629] virtio_net virtio0 ens2: renamed from eth0 [ 4.997489] scsi host0: ata_piix [ 5.001413] scsi host1: ata_piix [ 5.005544] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.008474] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 5.043927] random: fast init done [ 8.711554] dracut-initqueue[592]: RTNETLINK answers: File exists [ 9.864415] random: crng init done [ 9.865719] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.258976] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.407736] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.706160] SELinux: Disabled at runtime. [ 11.773771] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.782329] systemd[1]: Detected virtualization kvm. [ 11.784307] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.366866] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.369877] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.374744] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.378884] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.382199] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.388868] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.392233] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ 12.516696] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [[ 12.849563] squashfs: version 4.0 (2009/01/31) Phillip Lougher  OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.177833] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.233762] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.346507] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.370760] EDAC sbridge: Ver: 1.1.2 [ 15.132660] Key type dns_resolver registered [ 15.446111] NFS: Registering the id_resolver key type [ 15.448312] Key type id_resolver registered [ 15.449853] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg411-server login: [ 46.865369] libcfs: loading out-of-tree module taints kernel. [ 46.948989] Key type ._llcrypt registered [ 46.955110] Key type .llcrypt registered [ 47.145257] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 67.499052] hrtimer: interrupt took 5043211 ns [ 68.049549] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 70.194094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.210874] alg: No test for adler32 (adler32-zlib) [ 71.648936] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 72.875838] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 74.792847] Key type lgssc registered [ 77.029394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.411908] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 147.648386] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 161.348536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.401638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.707441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 162.740775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 162.880680] Lustre: lustre-MDT0000: new disk, initializing [ 163.015936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.039734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.126651] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.444784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.536274] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.575509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 182.580288] Lustre: Skipped 1 previous similar message [ 182.650269] Lustre: lustre-MDT0001: new disk, initializing [ 182.708052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 182.730462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 182.740722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.855168] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.545235] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.738262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.039373] Lustre: lustre-OST0000: new disk, initializing [ 202.044762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.064530] Lustre: 8456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.168023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.197672] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.500920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 211.521421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 211.605569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 223.139978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.302376] Lustre: lustre-OST0001: new disk, initializing [ 223.306707] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 223.316212] Lustre: 9530:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.403382] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.292573] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.922706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 230.932194] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 231.025953] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 242.908819] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 253.869875] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 261.453742] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing check_logdir /tmp/testlogs/ [ 266.544150] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing yml_node [ 272.333198] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 276.304535] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 279.005987] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 281.016510] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 13 08:28:26 EDT 2026 [ 299.316739] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 301.284147] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 317.925674] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.927239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.943900] Lustre: Skipped 1 previous similar message [ 323.087788] Lustre: server umount lustre-MDT0000 complete [ 327.514530] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302554 with bad export cookie 6402984941974445460 [ 327.517279] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 327.533164] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 328.162480] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.162582] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.179757] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 328.205357] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 328.218888] Lustre: Skipped 3 previous similar messages [ 328.224446] Lustre: Skipped 3 previous similar messages [ 332.201913] LustreError: 8351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.203406] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 332.245216] Lustre: Skipped 2 previous similar messages [ 333.932199] Lustre: server umount lustre-MDT0001 complete [ 338.283615] Lustre: server umount lustre-OST0000 complete [ 342.843412] Lustre: server umount lustre-OST0001 complete [ 350.818270] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 360.550913] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 401.183246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.693605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 401.724434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 401.819384] Lustre: lustre-MDT0000: new disk, initializing [ 401.893187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.911924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 407.552316] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 420.566789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.727182] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 420.846455] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 420.855966] Lustre: Skipped 1 previous similar message [ 421.001228] Lustre: lustre-MDT0001: new disk, initializing [ 421.095183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 421.146459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 421.179071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 426.739435] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 438.717655] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 444.998563] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 454.772259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 455.107534] Lustre: lustre-OST0000: new disk, initializing [ 455.113472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 455.123894] Lustre: 17051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 455.226028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 456.395470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 456.413206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 456.551478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 464.020904] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 475.022752] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 479.884939] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 480.128865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 485.462781] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 485.661603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 492.515337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.520342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 492.539225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 492.555495] Lustre: Skipped 1 previous similar message [ 493.772090] Lustre: server umount lustre-OST0000 complete [ 497.635556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.638101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 497.646308] Lustre: Skipped 2 previous similar messages [ 497.657455] Lustre: Skipped 1 previous similar message [ 503.527421] Lustre: server umount lustre-MDT0000 complete [ 507.399338] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302734 with bad export cookie 6402984941974446993 [ 507.402189] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 507.424495] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 507.800724] Lustre: server umount lustre-MDT0001 complete [ 511.819694] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 08:32:17 (1789302737) [ 519.173286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.597053] LustreError: 19315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.614513] LustreError: 19315:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 519.671918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 524.063192] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 524.773197] LustreError: 19316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.870574] LustreError: 19315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.382918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.059673] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 545.346412] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 552.618984] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 561.167919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 568.173689] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 575.706217] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 587.774186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 587.792233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.815881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 587.819021] Lustre: Skipped 2 previous similar messages [ 593.285889] Lustre: server umount lustre-OST0000 complete [ 598.499815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 598.530000] Lustre: Skipped 2 previous similar messages [ 608.740962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 608.746399] Lustre: Skipped 7 previous similar messages [ 612.319864] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 612.483993] Lustre: server umount lustre-MDT0000 complete [ 613.858336] LustreError: 19311:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.874115] LustreError: 19311:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 616.493694] LustreError: 19297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302843 with bad export cookie 6402984941974447868 [ 616.495581] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 616.507962] LustreError: 19297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 616.857986] Lustre: server umount lustre-MDT0001 complete [ 630.887518] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 634.518678] Key type lgssc unregistered [ 635.001212] LNet: 22849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 635.019828] LNetError: 22849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 635.070325] LNet: Removed LNI 192.168.204.111@tcp [ 636.597166] Key type .llcrypt unregistered [ 636.604381] Key type ._llcrypt unregistered [ 659.736851] Key type ._llcrypt registered [ 659.739491] Key type .llcrypt registered [ 659.883161] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 661.134321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 661.153638] alg: No test for adler32 (adler32-zlib) [ 662.188101] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 662.447790] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 664.119466] Key type lgssc registered [ 665.247545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 673.735330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 673.770388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 675.190087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.351143] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 685.538635] LustreError: 24255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 685.569109] LustreError: 24255:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 690.058650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.477624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 695.514804] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 703.859519] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 710.236331] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 717.153856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 717.664429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.763037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 723.597729] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 731.949674] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 750.421165] Lustre: server umount lustre-OST0000 complete [ 755.681075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.701705] Lustre: Skipped 1 previous similar message [ 755.711492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 755.718037] Lustre: Skipped 1 previous similar message [ 760.800993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 760.806628] Lustre: Skipped 1 previous similar message [ 764.997984] Lustre: server umount lustre-MDT0000 complete [ 765.919168] Lustre: 23565:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302977/real 1789302977] req@00000000f60b073f x1876220066166016/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789302993 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 765.919863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 765.924628] LustreError: 24254:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 765.958091] Lustre: 23565:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 769.057117] LustreError: 24254:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 769.086935] LustreError: 24254:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 769.171946] LustreError: 24235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302996 with bad export cookie 5060604723132908313 [ 769.184197] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 769.189139] LustreError: 24235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 769.575438] Lustre: server umount lustre-MDT0001 complete [ 781.173474] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 784.254435] Key type lgssc unregistered [ 784.537308] LNet: 28227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.555812] LNetError: 28227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.581774] LNet: Removed LNI 192.168.204.111@tcp [ 785.757886] Key type .llcrypt unregistered [ 785.761663] Key type ._llcrypt unregistered [ 809.357608] Key type ._llcrypt registered [ 809.359512] Key type .llcrypt registered [ 809.425350] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 810.290491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 810.499183] alg: No test for adler32 (adler32-zlib) [ 811.523676] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 811.725057] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 813.391230] Key type lgssc registered [ 814.529337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 821.925026] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 821.942391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.504900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.969177] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 832.997969] LustreError: 29637:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 833.011783] LustreError: 29637:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 838.112205] LustreError: 29638:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 839.200420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 839.662618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 844.954357] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 853.249469] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 857.917962] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 872.021413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 872.398016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 878.895624] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 886.581264] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 893.951581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 905.184817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 905.195674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 905.205594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 907.276744] Lustre: server umount lustre-OST0000 complete [ 914.400928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 914.411204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 914.429494] Lustre: Skipped 1 previous similar message [ 914.465363] Lustre: Skipped 1 previous similar message [ 919.523631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 919.538107] Lustre: Skipped 1 previous similar message [ 921.675826] Lustre: server umount lustre-MDT0000 complete [ 924.649569] LustreError: 31665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 924.675801] LustreError: 31665:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 926.030934] LustreError: 29616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303153 with bad export cookie 15582510995851761383 [ 926.045790] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 926.049966] LustreError: 29616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 926.405988] Lustre: server umount lustre-MDT0001 complete [ 937.843763] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 941.016209] Key type lgssc unregistered [ 941.334917] LNet: 33536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.343274] LNetError: 33536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.355445] LNet: Removed LNI 192.168.204.111@tcp [ 942.267797] Key type .llcrypt unregistered [ 942.270860] Key type ._llcrypt unregistered [ 967.693093] Key type ._llcrypt registered [ 967.698345] Key type .llcrypt registered [ 967.864390] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 969.150569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 969.218353] alg: No test for adler32 (adler32-zlib) [ 970.336364] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 970.629259] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 972.376153] Key type lgssc registered [ 973.735060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 982.637956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 982.659037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 984.112531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.940604] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 994.282378] LustreError: 34948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 994.305445] LustreError: 34948:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 998.730684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.271473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1004.839903] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1012.084052] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1017.570472] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1023.574454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1023.853925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.987202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1030.232672] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1037.837403] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1049.580644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1049.580817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.588251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1049.628856] Lustre: Skipped 1 previous similar message [ 1052.541531] Lustre: server umount lustre-OST0000 complete [ 1059.808865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.821344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1059.827528] Lustre: Skipped 2 previous similar messages [ 1063.043572] Lustre: server umount lustre-MDT0000 complete [ 1064.932850] LustreError: 34944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.955505] LustreError: 34944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1066.551279] LustreError: 34929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303293 with bad export cookie 10218760084502606631 [ 1066.553190] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1066.563326] LustreError: 34929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1066.950975] Lustre: server umount lustre-MDT0001 complete [ 1078.735369] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1081.569427] Key type lgssc unregistered [ 1081.891674] LNet: 38480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.904301] LNetError: 38480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.917264] LNet: Removed LNI 192.168.204.111@tcp [ 1082.710302] Key type .llcrypt unregistered [ 1082.712718] Key type ._llcrypt unregistered [ 1106.449538] Key type ._llcrypt registered [ 1106.452471] Key type .llcrypt registered [ 1106.568828] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1107.781643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1107.988991] alg: No test for adler32 (adler32-zlib) [ 1109.310891] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 1109.672952] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1111.439305] Key type lgssc registered [ 1112.855996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1120.829879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1120.867975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1122.378393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1127.494561] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1132.527419] LustreError: 39891:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1132.561181] LustreError: 39891:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1137.277222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.620391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1142.059780] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1151.365830] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1157.751837] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1165.494654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1165.921743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1167.990148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1172.700361] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1181.085492] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1188.835959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1188.839248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.853944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1188.872175] Lustre: Skipped 1 previous similar message [ 1188.896767] Lustre: Skipped 1 previous similar message [ 1193.962062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1193.972233] Lustre: Skipped 2 previous similar messages [ 1194.101184] LustreError: 41710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1194.221694] Lustre: server umount lustre-OST0000 complete [ 1218.917616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1218.920371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1218.935689] Lustre: Skipped 1 previous similar message [ 1224.161152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1224.174226] Lustre: Skipped 2 previous similar messages [ 1229.279187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1229.282029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1229.294714] Lustre: Skipped 1 previous similar message [ 1229.607424] Lustre: server umount lustre-MDT0000 complete [ 1233.755135] LustreError: 39873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303460 with bad export cookie 16084189556347473107 [ 1233.773205] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1248.223187] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1248.686653] Lustre: server umount lustre-MDT0001 complete [ 1260.029454] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1263.499754] Key type lgssc unregistered [ 1263.826487] LNet: 43601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.835552] LNetError: 43601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.849796] LNet: Removed LNI 192.168.204.111@tcp [ 1264.833096] Key type .llcrypt unregistered [ 1264.839145] Key type ._llcrypt unregistered [ 1290.921341] Key type ._llcrypt registered [ 1290.925792] Key type .llcrypt registered [ 1291.041271] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1292.075827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1292.168905] alg: No test for adler32 (adler32-zlib) [ 1293.278563] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 1293.532940] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1295.297372] Key type lgssc registered [ 1296.887873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1305.736805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1305.765115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1307.247575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1312.925989] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1317.861525] LustreError: 45010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1317.897484] LustreError: 45010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1322.886200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1322.977392] LustreError: 45011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.284236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1328.043649] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1336.169428] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1342.096858] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1348.524203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1348.793198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.906071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1356.621373] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1364.817434] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1377.901722] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 1378.790944] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.793022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1378.805191] Lustre: Skipped 1 previous similar message [ 1381.677429] Lustre: server umount lustre-MDT0000 complete [ 1383.006515] LustreError: 45007:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1385.550895] LustreError: 44991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303612 with bad export cookie 16648573441983427742 [ 1385.558701] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1385.565105] LustreError: 44991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1388.124052] LustreError: 45007:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.134621] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [ 1388.141328] LustreError: 45007:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1388.176768] Lustre: Skipped 2 previous similar messages [ 1389.027781] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.047924] Lustre: Skipped 1 previous similar message [ 1392.074040] Lustre: server umount lustre-MDT0001 complete [ 1408.119921] Lustre: server umount lustre-OST0000 complete [ 1429.030803] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1432.933823] Key type lgssc unregistered [ 1433.319043] LNet: 48873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1433.345659] LNetError: 48873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1433.382686] LNet: Removed LNI 192.168.204.111@tcp [ 1434.398798] Key type .llcrypt unregistered [ 1434.401870] Key type ._llcrypt unregistered [ 1462.663318] Key type ._llcrypt registered [ 1462.666680] Key type .llcrypt registered [ 1462.814081] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1463.934337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1464.067866] alg: No test for adler32 (adler32-zlib) [ 1465.298249] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 1465.581672] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1467.359400] Key type lgssc registered [ 1468.669030] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1477.094197] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1477.128805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1492.768346] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1497.951381] LustreError: 50258:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 1524.641259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1531.490928] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1539.103499] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1706.773506] Lustre: server umount lustre-OST0000 complete [ 1723.676302] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1726.741137] Key type lgssc unregistered [ 1727.193325] LNet: 51933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.200676] LNetError: 51933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1728.230562] LNet: Removed LNI 192.168.204.111@tcp [ 1729.170984] Key type .llcrypt unregistered [ 1729.179843] Key type ._llcrypt unregistered [ 1755.523136] Key type ._llcrypt registered [ 1755.525176] Key type .llcrypt registered [ 1755.626415] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1756.804139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1756.959271] alg: No test for adler32 (adler32-zlib) [ 1758.102588] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 1758.387451] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1760.079582] Key type lgssc registered [ 1761.269310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1770.126556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1770.149995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.657142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.447955] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1781.218809] LustreError: 53344:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1781.241248] LustreError: 53344:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1785.777084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.280402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1791.408154] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1799.983076] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1806.220181] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1813.859496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1814.264832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.316991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1821.362605] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1829.335425] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1837.029187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1837.031251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1837.044128] LustreError: Skipped 1 previous similar message [ 1837.047069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1837.069026] Lustre: Skipped 1 previous similar message [ 1838.929065] Lustre: server umount lustre-OST0000 complete [ 1847.265395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1847.279148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1847.291859] Lustre: Skipped 1 previous similar message [ 1847.306833] Lustre: Skipped 2 previous similar messages [ 1849.785832] Lustre: server umount lustre-MDT0000 complete [ 1852.384396] LustreError: 53345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1852.415278] LustreError: 53345:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1854.415697] LustreError: 53325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304081 with bad export cookie 7484903925429931149 [ 1854.422400] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1854.426037] LustreError: 53325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1854.907429] Lustre: server umount lustre-MDT0001 complete [ 1866.460515] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1870.004449] Key type lgssc unregistered [ 1870.397721] LNet: 56877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.407992] LNetError: 56877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1870.427911] LNet: Removed LNI 192.168.204.111@tcp [ 1871.381979] Key type .llcrypt unregistered [ 1871.384875] Key type ._llcrypt unregistered [ 1897.294404] Key type ._llcrypt registered [ 1897.298990] Key type .llcrypt registered [ 1897.416532] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1899.027261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1899.049858] alg: No test for adler32 (adler32-zlib) [ 1900.135481] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 1900.606461] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1902.439297] Key type lgssc registered [ 1904.087395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1913.877420] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1913.930759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1929.695475] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1934.815480] LustreError: 58260:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 1961.567158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.093593] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1978.917292] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1986.818371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.322648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.395821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1992.176824] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1992.682039] LustreError: 58284:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1992.714606] LustreError: 58284:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1996.774411] LustreError: 59300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.795974] LustreError: 59300:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2001.538575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.040826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2007.466274] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2016.912405] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2023.054258] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2027.489762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2027.509895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2027.528782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2031.494786] Lustre: server umount lustre-OST0000 complete [ 2055.475736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.510086] Lustre: Skipped 2 previous similar messages [ 2055.527076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.547246] Lustre: Skipped 1 previous similar message [ 2060.768647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.781069] Lustre: Skipped 2 previous similar messages [ 2062.065768] Lustre: server umount lustre-MDT0000 complete [ 2065.896020] LustreError: 59304:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.909190] LustreError: 59304:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2066.542694] LustreError: 58268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304293 with bad export cookie 2654570363778132194 [ 2066.549022] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2066.553058] LustreError: 58268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2073.208801] Lustre: server umount lustre-MDT0001 complete [ 2085.428674] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2088.696667] Key type lgssc unregistered [ 2088.962887] LNet: 62005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.968478] LNetError: 62005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2088.984651] LNet: Removed LNI 192.168.204.111@tcp [ 2089.907333] Key type .llcrypt unregistered [ 2089.913343] Key type ._llcrypt unregistered [ 2117.983430] Key type ._llcrypt registered [ 2117.986711] Key type .llcrypt registered [ 2118.100952] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2119.264065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2119.446352] alg: No test for adler32 (adler32-zlib) [ 2120.619419] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 2120.988067] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2122.737177] Key type lgssc registered [ 2124.292756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2134.295503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2134.313585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.951810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.210192] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2146.277352] LustreError: 63418:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.308581] LustreError: 63418:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2151.312556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.399311] LustreError: 63417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.852555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2157.392402] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2167.293952] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2174.101551] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2182.324611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2182.866482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2190.905485] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2201.663479] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2206.559214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2222.075289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2222.079130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.082337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2222.094453] LustreError: Skipped 1 previous similar message [ 2222.122742] Lustre: Skipped 1 previous similar message [ 2227.184897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2227.191059] Lustre: Skipped 2 previous similar messages [ 2232.296311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2232.304883] Lustre: Skipped 1 previous similar message [ 2234.335127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2234.979951] Lustre: server umount lustre-OST0000 complete [ 2242.529122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.531080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.550602] Lustre: Skipped 1 previous similar message [ 2242.560194] Lustre: Skipped 1 previous similar message [ 2247.650680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.666389] Lustre: Skipped 1 previous similar message [ 2254.305136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2254.734533] Lustre: server umount lustre-MDT0000 complete [ 2257.899169] LustreError: 65911:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.918287] LustreError: 65911:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2259.277469] LustreError: 63397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304486 with bad export cookie 18053743640393442801 [ 2259.293602] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2259.299342] LustreError: 63397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2259.672574] Lustre: server umount lustre-MDT0001 complete [ 2273.690301] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2276.487859] Key type lgssc unregistered [ 2276.900766] LNet: 66990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.909963] LNetError: 66990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.922819] LNet: Removed LNI 192.168.204.111@tcp [ 2277.987185] Key type .llcrypt unregistered [ 2277.989614] Key type ._llcrypt unregistered [ 2305.571670] Key type ._llcrypt registered [ 2305.574292] Key type .llcrypt registered [ 2305.717190] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2306.847604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2306.871773] alg: No test for adler32 (adler32-zlib) [ 2307.999867] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 2308.243255] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2309.927396] Key type lgssc registered [ 2310.934292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2322.114157] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:02:27 (1789304547) [ 2329.715441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2329.755562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.561233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.657736] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2340.900964] LustreError: 68691:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.926922] LustreError: 68691:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2345.961507] LustreError: 68692:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.049495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.436641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2354.482357] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2363.458065] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2370.416850] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2377.875652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2378.492970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2385.042608] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2394.283375] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2401.012140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2405.566368] Lustre: Failing over lustre-MDT0000 [ 2405.857408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2405.865566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.875213] LustreError: 68692:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.369670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2406.383863] Lustre: Skipped 1 previous similar message [ 2407.832307] Lustre: server umount lustre-MDT0000 complete [ 2411.490526] LustreError: 71013:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.514704] LustreError: 71013:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2416.613446] LustreError: 68692:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.637809] LustreError: 68692:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2417.684899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2417.817114] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2418.031640] Lustre: *** cfs_fail_loc=135, val=0*** [ 2418.175575] LustreError: 71780:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2418.186772] LustreError: 71780:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.111@tcp: cfg command failed: rc = -2 [ 2418.196722] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2418.196722] [ 2418.206228] LustreError: MGC192.168.204.111@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2418.227854] LustreError: 71769:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2418.234708] LustreError: 71769:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2418.243098] LustreError: 71769:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2418.248676] LustreError: 71769:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2418.277975] Lustre: server umount lustre-MDT0000 complete [ 2418.281993] LustreError: 71769:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2425.963391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2426.113840] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2426.322282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2426.361918] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2426.986646] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2431.480120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2431.531202] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2431.590700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2431.776511] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2441.186045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2441.193145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.205426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2444.263887] Lustre: server umount lustre-OST0000 complete [ 2451.937052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2451.940773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.941962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2451.955796] Lustre: Skipped 2 previous similar messages [ 2451.967195] Lustre: Skipped 2 previous similar messages [ 2457.066319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2457.075469] Lustre: Skipped 1 previous similar message [ 2461.157772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2461.164325] Lustre: Skipped 1 previous similar message [ 2462.177013] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2462.407519] Lustre: server umount lustre-MDT0000 complete [ 2466.282639] LustreError: 68687:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.297052] LustreError: 68687:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2466.467534] LustreError: 68674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304693 with bad export cookie 3688751217620128435 [ 2466.479567] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.482454] LustreError: 68674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2466.889462] Lustre: server umount lustre-MDT0001 complete [ 2476.721645] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2479.691016] Key type lgssc unregistered [ 2479.993932] LNet: 73561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2480.006941] LNetError: 73561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2480.025608] LNet: Removed LNI 192.168.204.111@tcp [ 2480.819075] Key type .llcrypt unregistered [ 2480.822518] Key type ._llcrypt unregistered [ 2503.467380] Key type ._llcrypt registered [ 2503.474076] Key type .llcrypt registered [ 2503.556381] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2504.567601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2504.694549] alg: No test for adler32 (adler32-zlib) [ 2505.808797] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 2506.081686] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2507.847354] Key type lgssc registered [ 2509.149693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2516.060561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2516.072360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.549730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2521.787611] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2527.720511] LustreError: 74968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.749661] LustreError: 74968:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2531.119588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.659459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2536.259383] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2544.697543] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2550.517087] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2557.469590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2557.951479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.043219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2564.145924] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2572.123896] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2581.771226] Lustre: Failing over lustre-MDT0000 [ 2582.125238] Lustre: server umount lustre-MDT0000 complete [ 2583.008095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2583.015223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2583.038674] LustreError: 74969:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.225542] LustreError: 76508:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.594144] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2586.613878] Lustre: Skipped 1 previous similar message [ 2590.290648] LustreError: 74963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.311221] LustreError: 74963:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2591.494376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2591.616441] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2591.716186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2591.722599] Lustre: Skipped 2 previous similar messages [ 2591.790071] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2591.914485] LustreError: 78056:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2591.922119] LustreError: 78056:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.111@tcp: cfg command failed: rc = -9 [ 2591.938037] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2591.938037] [ 2591.947109] LustreError: MGC192.168.204.111@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2591.961671] LustreError: 78044:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2591.968843] LustreError: 78044:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2591.987796] LustreError: 78044:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2591.995278] LustreError: 78044:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2592.018890] Lustre: server umount lustre-MDT0000 complete [ 2592.021576] LustreError: 78044:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2595.413721] LustreError: 74964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.937040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.104951] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2598.350204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2598.411726] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2600.534050] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2602.642932] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2603.500609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2603.531112] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2603.622408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2612.192365] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2612.196769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.214241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2613.736215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2613.747910] Lustre: Skipped 1 previous similar message [ 2616.145985] Lustre: server umount lustre-OST0000 complete [ 2623.968774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2623.979345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2623.982644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2623.999994] Lustre: Skipped 2 previous similar messages [ 2629.092054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2629.100779] Lustre: Skipped 2 previous similar messages [ 2634.208544] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2634.412329] Lustre: server umount lustre-MDT0000 complete [ 2638.306756] LustreError: 75697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.341356] LustreError: 75697:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2638.452196] LustreError: 74950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304865 with bad export cookie 3855728054630782881 [ 2638.454593] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2638.460813] LustreError: 74950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2638.878410] Lustre: server umount lustre-MDT0001 complete [ 2649.542519] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2652.420056] Key type lgssc unregistered [ 2652.793814] LNet: 79835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2652.805599] LNetError: 79835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2652.826928] LNet: Removed LNI 192.168.204.111@tcp [ 2653.767965] Key type .llcrypt unregistered [ 2653.772122] Key type ._llcrypt unregistered [ 2665.580386] Key type ._llcrypt registered [ 2665.583760] Key type .llcrypt registered [ 2665.703308] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2665.932192] Key type .llcrypt unregistered [ 2665.934954] Key type ._llcrypt unregistered [ 2686.150135] Key type ._llcrypt registered [ 2686.151405] Key type .llcrypt registered [ 2686.224732] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2687.279608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2687.304451] alg: No test for adler32 (adler32-zlib) [ 2688.326954] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 2688.507795] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2690.159205] Key type lgssc registered [ 2691.110983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2698.151135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2698.164784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.556642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.704788] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2708.965593] LustreError: 81707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.977257] LustreError: 81707:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2711.884602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.282520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2716.390835] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2722.605618] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2727.793630] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2733.800501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2734.163306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2740.099187] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2744.467196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2747.579964] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2758.118851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2758.124375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.140269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2760.172540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2760.172635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.206500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2762.050769] Lustre: server umount lustre-OST0000 complete [ 2768.869674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2768.880079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2768.909203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2768.923202] Lustre: Skipped 1 previous similar message [ 2772.033986] Lustre: server umount lustre-MDT0000 complete [ 2775.193867] LustreError: 84201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305002 with bad export cookie 8576966247700930791 [ 2775.200289] LustreError: 84201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2775.201590] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2775.467561] Lustre: server umount lustre-MDT0001 complete [ 2784.347495] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2786.617871] Key type lgssc unregistered [ 2786.899510] LNet: 85239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2786.908853] LNetError: 85239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2786.920985] LNet: Removed LNI 192.168.204.111@tcp [ 2787.722725] Key type .llcrypt unregistered [ 2787.725964] Key type ._llcrypt unregistered [ 2807.257258] Key type ._llcrypt registered [ 2807.259206] Key type .llcrypt registered [ 2807.340781] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2808.255329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2808.285332] alg: No test for adler32 (adler32-zlib) [ 2809.272688] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 2809.471552] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2811.119209] Key type lgssc registered [ 2812.100368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2818.746972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2818.767804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.120375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2823.901514] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2830.307883] LustreError: 86625:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.322910] LustreError: 86625:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2831.720799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.022181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2835.730049] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2841.919872] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2846.959817] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2852.299463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2852.551617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.547937] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2863.615309] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2865.852125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2876.394265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.399130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2876.420711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2876.425128] Lustre: Skipped 1 previous similar message [ 2879.683857] Lustre: server umount lustre-OST0000 complete [ 2883.041554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2883.050714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2883.060990] Lustre: Skipped 1 previous similar message [ 2883.064957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2886.624462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2886.626664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.606569] Lustre: server umount lustre-MDT0000 complete [ 2891.281748] LustreError: 86608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305118 with bad export cookie 17447883186196259390 [ 2891.285797] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2891.292333] LustreError: 86608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2891.507485] Lustre: server umount lustre-MDT0001 complete [ 2899.967211] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2902.129411] Key type lgssc unregistered [ 2902.361858] LNet: 90158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2902.367755] LNetError: 90158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2902.383525] LNet: Removed LNI 192.168.204.111@tcp [ 2903.069182] Key type .llcrypt unregistered [ 2903.071676] Key type ._llcrypt unregistered [ 2921.407879] Key type ._llcrypt registered [ 2921.410691] Key type .llcrypt registered [ 2921.508813] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2922.228060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2922.301799] alg: No test for adler32 (adler32-zlib) [ 2923.255828] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 2923.427772] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2925.055684] Key type lgssc registered [ 2925.815819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2931.450035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2931.465818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.720113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2935.640489] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2941.691618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.924570] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2942.018519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2945.500394] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2951.470109] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2955.759380] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2961.821364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2962.045716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2966.406146] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2971.070487] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2977.392423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2991.076750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2991.086043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.098627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2993.120862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2993.123888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2993.130104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2994.420085] Lustre: server umount lustre-OST0000 complete [ 2998.243881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2998.248492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.256779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2998.256787] Lustre: Skipped 1 previous similar message [ 2998.292983] Lustre: Skipped 1 previous similar message [ 3003.360057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3003.364996] Lustre: Skipped 2 previous similar messages [ 3003.591391] Lustre: server umount lustre-MDT0000 complete [ 3007.009452] LustreError: 91524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305234 with bad export cookie 10168704650327095740 [ 3007.016590] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3007.029539] LustreError: 91524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3007.248259] Lustre: server umount lustre-MDT0001 complete [ 3014.819814] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3017.036253] Key type lgssc unregistered [ 3017.284758] LNet: 95098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3017.289620] LNetError: 95098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3017.301710] LNet: Removed LNI 192.168.204.111@tcp [ 3017.913970] Key type .llcrypt unregistered [ 3017.916558] Key type ._llcrypt unregistered [ 3035.055377] Key type ._llcrypt registered [ 3035.057394] Key type .llcrypt registered [ 3035.125269] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3036.129554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3036.174629] alg: No test for adler32 (adler32-zlib) [ 3037.179757] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 3037.415813] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3039.079210] Key type lgssc registered [ 3039.940807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3046.200540] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3046.215996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3061.727608] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3066.847448] LustreError: 96458:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 3093.471207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.384743] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3101.924789] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3105.788653] Lustre: server umount lustre-OST0000 complete [ 3111.530787] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:15:37 (1789305337) [ 3116.173890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.558730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.541395] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3121.633891] LustreError: 98092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3121.652261] LustreError: 98092:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3126.108340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.383064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3129.209447] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3134.454277] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3138.501511] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3143.255091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3143.542230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.554267] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3152.424862] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3161.964772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3172.325277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3172.325438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3172.333180] LustreError: Skipped 1 previous similar message [ 3172.342105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3172.348911] Lustre: Skipped 1 previous similar message [ 3175.254429] Lustre: server umount lustre-OST0000 complete [ 3182.563876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3182.569261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3182.572056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3182.583136] Lustre: Skipped 2 previous similar messages [ 3187.685357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3187.689297] Lustre: Skipped 2 previous similar messages [ 3192.287163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3192.439946] Lustre: server umount lustre-MDT0000 complete [ 3192.801305] LustreError: 100583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3192.821978] LustreError: 100583:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3195.491261] LustreError: 98073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305422 with bad export cookie 16031611067120017014 [ 3195.494872] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3195.500825] LustreError: 98073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3195.753646] Lustre: server umount lustre-MDT0001 complete [ 3204.496459] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3206.501928] Key type lgssc unregistered [ 3206.713749] LNet: 101814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3206.717952] LNetError: 101814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3207.793644] LNet: Removed LNI 192.168.204.111@tcp [ 3208.328858] Key type .llcrypt unregistered [ 3208.331367] Key type ._llcrypt unregistered [ 3240.150522] Key type ._llcrypt registered [ 3240.155814] Key type .llcrypt registered [ 3240.234619] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3240.931177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3240.989213] alg: No test for adler32 (adler32-zlib) [ 3241.958826] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 3242.125749] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3243.759669] Key type lgssc registered [ 3244.515401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3249.974522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3249.998931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3251.306121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.875471] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3259.520592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.757602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3262.418565] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3267.561659] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3271.322940] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3275.751055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3275.999886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.034231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3279.482193] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3284.163830] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3292.640713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3292.645727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3292.652748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3294.691393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3294.692108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3294.702364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3295.818813] Lustre: server umount lustre-OST0000 complete [ 3299.810942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.811899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.831094] Lustre: Skipped 1 previous similar message [ 3299.831101] Lustre: Skipped 1 previous similar message [ 3304.388085] Lustre: server umount lustre-MDT0000 complete [ 3304.929044] LustreError: 103973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.942478] LustreError: 103973:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3306.540723] LustreError: 104706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305533 with bad export cookie 2337994414454405335 [ 3306.547459] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3306.552037] LustreError: 104706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3306.759933] Lustre: server umount lustre-MDT0001 complete [ 3312.765991] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3314.543031] Key type lgssc unregistered [ 3314.755671] LNet: 107498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3314.762924] LNetError: 107498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3314.776342] LNet: Removed LNI 192.168.204.111@tcp [ 3315.298682] Key type .llcrypt unregistered [ 3315.300380] Key type ._llcrypt unregistered [ 3327.846313] Key type ._llcrypt registered [ 3327.847470] Key type .llcrypt registered [ 3327.915826] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3328.546803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3328.633152] alg: No test for adler32 (adler32-zlib) [ 3329.639380] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 3329.821308] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3331.463221] Key type lgssc registered [ 3332.266129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3338.122889] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3338.137261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3353.633137] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3358.751363] LustreError: 108716:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 3385.375493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3389.216899] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3394.426897] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3398.888191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.057119] LustreError: MGC192.168.204.111@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3399.065226] LustreError: 109740:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3399.069493] LustreError: 109740:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3399.074628] LustreError: 109740:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3399.133518] Lustre: server umount lustre-MDT0000 complete [ 3399.135934] LustreError: 109740:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3414.555609] Lustre: server umount lustre-OST0000 complete [ 3419.866484] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 3425.823211] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3449.919466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3450.063924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3450.087647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3450.137510] Lustre: lustre-MDT0000: new disk, initializing [ 3450.181466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3450.198989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3452.840169] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3459.899888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3459.961633] Lustre: 112527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3460.008612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3460.014248] Lustre: Skipped 1 previous similar message [ 3460.058207] Lustre: lustre-MDT0001: new disk, initializing [ 3460.120699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3460.149310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3460.154083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3462.453572] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3468.147176] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3471.536707] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3475.229525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3475.393476] Lustre: lustre-OST0000: new disk, initializing [ 3475.397331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3475.403642] Lustre: 114497:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3475.456513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.385219] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3481.095085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3481.102540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3481.150315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3483.349935] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3485.451191] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3485.556744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3487.902530] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3488.025954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3491.296420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3491.301377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3491.302955] LustreError: Skipped 1 previous similar message [ 3491.304549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3491.315338] Lustre: Skipped 1 previous similar message [ 3491.325504] Lustre: Skipped 1 previous similar message [ 3495.154812] Lustre: server umount lustre-OST0000 complete [ 3501.025983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3501.030266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.035720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3503.285634] Lustre: server umount lustre-MDT0000 complete [ 3505.403626] LustreError: 112521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305732 with bad export cookie 409720354130820524 [ 3505.408341] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3505.408723] LustreError: 112521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3505.622700] Lustre: server umount lustre-MDT0001 complete [ 3510.670248] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:22:17 (1789305737) [ 3512.736329] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3521.179583] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 3526.016902] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3545.239036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3545.351144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3545.368835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3545.419101] Lustre: lustre-MDT0000: new disk, initializing [ 3545.457855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3545.467360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3547.889654] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3553.995075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3554.057627] Lustre: 119389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3554.098322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3554.101385] Lustre: Skipped 1 previous similar message [ 3554.142121] Lustre: lustre-MDT0001: new disk, initializing [ 3554.195929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3554.237277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3554.248436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3556.309319] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3561.961704] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3565.182154] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3569.284600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3569.458699] Lustre: lustre-OST0000: new disk, initializing [ 3569.463404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3569.472378] Lustre: 121354:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3573.283386] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3577.372350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3577.382743] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3577.466772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3579.255107] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3582.113231] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3582.248917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3584.930937] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3585.077560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3587.552715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3587.555737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.561426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3587.566564] Lustre: Skipped 2 previous similar messages [ 3587.578219] Lustre: Skipped 2 previous similar messages [ 3592.315631] Lustre: server umount lustre-OST0000 complete [ 3595.232727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3595.239646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3595.251609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3600.729540] Lustre: server umount lustre-MDT0000 complete [ 3602.918630] LustreError: 119382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305830 with bad export cookie 409720354130821406 [ 3602.918974] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3602.926988] LustreError: 119382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3603.140763] Lustre: server umount lustre-MDT0001 complete [ 3608.634871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.933084] LustreError: 123568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3608.946109] LustreError: 123568:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3608.987474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3608.994486] Lustre: Skipped 1 previous similar message [ 3611.927987] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3614.178656] LustreError: 123569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3617.177749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.013589] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3624.763156] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3628.326184] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3632.536680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3636.111475] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3640.602259] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3648.927369] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3652.577301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3652.580059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3652.583890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3652.590280] Lustre: Skipped 2 previous similar messages [ 3652.597053] Lustre: Skipped 3 previous similar messages [ 3657.157669] Lustre: server umount lustre-OST0000 complete [ 3662.819593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3662.822075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3662.826660] Lustre: Skipped 1 previous similar message [ 3662.835497] Lustre: Skipped 1 previous similar message [ 3665.539486] Lustre: server umount lustre-MDT0000 complete [ 3667.696812] LustreError: 123550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305894 with bad export cookie 409720354130822288 [ 3667.699788] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3667.705496] LustreError: 123550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3667.916583] Lustre: server umount lustre-MDT0001 complete [ 3674.829966] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3676.555313] Key type lgssc unregistered [ 3676.767738] LNet: 127196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3676.774457] LNetError: 127196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3676.795016] LNet: Removed LNI 192.168.204.111@tcp [ 3677.393329] Key type .llcrypt unregistered [ 3677.395577] Key type ._llcrypt unregistered [ 3688.069345] Key type ._llcrypt registered [ 3688.071117] Key type .llcrypt registered [ 3688.134391] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 3696.993900] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3697.449716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3697.526816] alg: No test for adler32 (adler32-zlib) [ 3698.468390] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 3698.619036] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3700.247169] Key type lgssc registered [ 3700.964839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3722.857601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3722.867614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.021059] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3724.037277] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3724.090863] Lustre: lustre-MDT0000: new disk, initializing [ 3724.150142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3724.164285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3726.701608] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3733.329455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3733.405598] Lustre: 130321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3733.455971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3733.459901] Lustre: Skipped 1 previous similar message [ 3733.509904] Lustre: lustre-MDT0001: new disk, initializing [ 3733.566995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3733.595673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3733.601312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3735.909566] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3741.626732] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3744.660076] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3748.366637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3748.526533] Lustre: lustre-OST0000: new disk, initializing [ 3748.532645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3748.537636] Lustre: 132290:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3748.596364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3749.402506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3749.410278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3749.462899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3751.746153] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3757.257198] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3759.595695] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3759.708817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3761.700053] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3761.798831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3764.708441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3764.708880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.712482] LustreError: Skipped 1 previous similar message [ 3764.724324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3764.729684] Lustre: Skipped 1 previous similar message [ 3768.804452] Lustre: server umount lustre-OST0000 complete [ 3774.433931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3774.439689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3774.445855] Lustre: Skipped 1 previous similar message [ 3774.448420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3776.822591] Lustre: server umount lustre-MDT0000 complete [ 3778.668304] LustreError: 131116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306005 with bad export cookie 261512267930732173 [ 3778.670147] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3778.676814] LustreError: 131116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3778.865305] Lustre: server umount lustre-MDT0001 complete [ 3783.462571] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 09:26:49 (1789306009) [ 3786.979104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3787.244289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3789.398522] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3792.352976] LustreError: 134699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3792.360873] LustreError: 134699:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3793.903740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3794.143966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3796.407922] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3799.935336] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3802.944539] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3803.990345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.991162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3803.995347] Lustre: Skipped 2 previous similar messages [ 3804.001547] Lustre: Skipped 2 previous similar messages [ 3810.156570] Lustre: server umount lustre-MDT0000 complete [ 3810.273631] LustreError: 134693:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.281123] LustreError: 134693:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3811.942913] LustreError: 134678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306039 with bad export cookie 261512267930733048 [ 3811.945995] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3811.954172] LustreError: 134678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3812.125864] Lustre: server umount lustre-MDT0001 complete [ 3816.304388] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 09:27:22 (1789306042) [ 3819.725068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3835.231654] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3840.351498] LustreError: 137109:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 3865.951281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3868.629965] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3872.107281] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3873.248497] Lustre: server umount lustre-OST0000 complete [ 3877.334678] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 09:28:23 (1789306103) [ 3880.814520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3881.079221] LustreError: 138553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3883.291153] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3886.561915] LustreError: 138554:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3887.873742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3888.092190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3888.096322] Lustre: Skipped 1 previous similar message [ 3890.411611] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3894.862725] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3898.029858] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3901.753274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3904.835456] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3908.375186] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3918.306621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3918.309936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3918.313071] Lustre: Skipped 2 previous similar messages [ 3918.320474] Lustre: Skipped 4 previous similar messages [ 3922.988181] Lustre: server umount lustre-MDT0000 complete [ 3923.425330] LustreError: 139605:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3923.432648] LustreError: 139605:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3924.602182] LustreError: 138533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306151 with bad export cookie 261512267930733699 [ 3924.607445] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3924.610084] LustreError: 138533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3924.745320] Lustre: server umount lustre-MDT0001 complete [ 3936.681971] Lustre: server umount lustre-OST0000 complete [ 3940.645827] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 09:29:27 (1789306167) [ 3943.615323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3943.832617] LustreError: 142199:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3943.871410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3943.876331] Lustre: Skipped 1 previous similar message [ 3945.675570] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3949.363215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3951.190811] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3954.191441] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3956.407423] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3959.429236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3962.448493] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3966.067049] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3968.362805] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3968.872377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3969.520729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3971.764392] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3971.868541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3974.115021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3974.115503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3974.116261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3974.127393] Lustre: Skipped 1 previous similar message [ 3978.971280] Lustre: server umount lustre-OST0000 complete [ 3984.355285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.359306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3984.363643] Lustre: Skipped 1 previous similar message [ 3984.372853] Lustre: Skipped 2 previous similar messages [ 3986.791707] Lustre: server umount lustre-MDT0000 complete [ 3988.500650] LustreError: 142181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306215 with bad export cookie 261512267930734756 [ 3988.503967] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3988.512078] LustreError: 142181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3988.637707] Lustre: server umount lustre-MDT0001 complete [ 3992.613618] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 09:30:19 (1789306219) [ 3996.214258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4011.744136] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4016.863477] LustreError: 146360:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 4042.528044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4042.531443] Lustre: Skipped 2 previous similar messages [ 4044.983493] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4047.931386] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4050.992814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.199248] LustreError: 146385:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4051.208523] LustreError: 146385:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4051.262754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4053.125993] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4056.953112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.180183] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4063.060889] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4066.200750] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4068.233898] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4068.342523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4070.431055] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4070.585980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4072.416768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4072.419892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.427731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4077.796944] Lustre: server umount lustre-MDT0000 complete [ 4079.585356] LustreError: 146367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306306 with bad export cookie 261512267930735771 [ 4079.587855] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4079.589874] LustreError: 146367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4079.763832] Lustre: server umount lustre-MDT0001 complete [ 4091.624256] Lustre: server umount lustre-OST0000 complete [ 4095.298821] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:32:01 (1789306321) [ 4098.322736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4113.824991] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4118.943390] LustreError: 150550:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 4147.549868] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4151.692243] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4155.376506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4155.636094] LustreError: 150575:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4155.653511] LustreError: 150575:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4155.716055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4157.943742] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4162.378159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4164.709309] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4168.226817] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4170.592836] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4173.670313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4173.750755] Lustre: lustre-OST0001: new disk, initializing [ 4173.753445] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4173.757249] Lustre: 153421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4173.800149] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4173.803199] Lustre: Skipped 5 previous similar messages [ 4176.646738] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4177.160912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4177.165841] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4177.214128] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4181.342100] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4183.212544] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4183.305729] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4185.074504] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4185.168703] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4186.593536] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4186.598079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4186.604415] Lustre: Skipped 2 previous similar messages [ 4186.607160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4186.610308] Lustre: Skipped 3 previous similar messages [ 4192.141953] Lustre: server umount lustre-OST0000 complete [ 4197.860165] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4197.863395] LustreError: Skipped 1 previous similar message [ 4200.225158] Lustre: server umount lustre-OST0001 complete [ 4216.287125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4216.365252] Lustre: server umount lustre-MDT0000 complete [ 4217.924908] LustreError: 150556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306445 with bad export cookie 261512267930736723 [ 4217.929604] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4217.931701] LustreError: 150556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4218.102573] Lustre: server umount lustre-MDT0001 complete [ 4227.632893] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 09:34:14 (1789306454) [ 4228.410752] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4229.115724] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4229.848459] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 09:34:16 (1789306456) [ 4232.912744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.089478] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4233.105569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4235.042071] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4238.851492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.915883] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4238.931765] Lustre: 156974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4240.874313] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4243.903219] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4246.243453] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4249.056819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4249.167238] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4250.287464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4251.935324] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4255.065515] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4256.905038] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4256.988116] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4258.685949] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4258.778432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4260.834905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4260.837368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4260.846764] LustreError: Skipped 1 previous similar message [ 4260.852168] Lustre: Skipped 5 previous similar messages [ 4260.856151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4260.861910] Lustre: Skipped 11 previous similar messages [ 4265.760161] Lustre: server umount lustre-OST0000 complete [ 4280.685392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4281.895765] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4281.900871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4281.901455] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4281.908588] Lustre: Skipped 1 previous similar message [ 4283.354913] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4286.503498] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4288.547570] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4288.637857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4290.342436] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4290.431490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4292.791559] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff958786532000.ost_server_uuid 50 [ 4293.497187] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff958786532000.ost_server_uuid in FULL state after 0 sec [ 4293.567344] LustreError: 156983:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.11@tcp) returned error from blocking AST (req@0000000004a9ceb5 x1876223249816576 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000065469c5e/0x3a11405aa6b219f lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.11@tcp remote: 0xbfa86dac94b67967 expref: 5 pid: 156981 timeout: 4393 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4293.582764] LustreError: lustre-MDT0000: A client on nid 192.168.204.11@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4293.589762] LustreError: 156971:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.11@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000065469c5e/0x3a11405aa6b219f lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.11@tcp remote: 0xbfa86dac94b67967 expref: 6 pid: 156981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4297.184248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4301.423099] Lustre: server umount lustre-OST0000 complete [ 4309.514559] Lustre: server umount lustre-MDT0000 complete [ 4311.299509] LustreError: 156967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306538 with bad export cookie 261512267930737906 [ 4311.301926] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4311.306184] LustreError: 156967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4317.520234] Lustre: server umount lustre-MDT0001 complete [ 4322.135371] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4323.406280] Key type lgssc unregistered [ 4323.548674] LNet: 163079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4323.551894] LNetError: 163079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4323.560434] LNet: Removed LNI 192.168.204.111@tcp [ 4323.966562] Key type .llcrypt unregistered [ 4323.968383] Key type ._llcrypt unregistered [ 4334.583182] Key type ._llcrypt registered [ 4334.584031] Key type .llcrypt registered [ 4334.627108] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4335.142594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4335.158334] alg: No test for adler32 (adler32-zlib) [ 4336.050401] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 4336.166362] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4337.767218] Key type lgssc registered [ 4338.231798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4341.354765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4341.365534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4342.563223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4344.225335] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4347.992518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.188778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4349.931277] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4352.943330] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4355.308045] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4358.149487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4358.340828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4360.817813] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4364.004240] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4367.595836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4371.665733] Lustre: Failing over lustre-MDT0000 [ 4371.782163] Lustre: server umount lustre-MDT0000 complete [ 4372.963033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4372.964301] LustreError: 166952:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4372.970129] Lustre: Skipped 2 previous similar messages [ 4372.986491] LustreError: 166952:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4378.080213] LustreError: 166951:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.088547] LustreError: 166951:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4382.939870] LustreError: 166305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.11@tcp arrived at 1789306610 with bad export cookie 5847874358942602158 [ 4382.945041] LustreError: 166305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4383.200191] LustreError: 164465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.206654] LustreError: 164465:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4386.377339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4386.466170] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4386.592742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4386.621692] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4388.435668] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4389.440023] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4389.445628] Lustre: lustre-MDT0000: Denying connection for new client 33f25bf8-c8bf-43af-b3e8-56fbc44ade0b (at 192.168.204.11@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 4391.916839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4397.026310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.028577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4397.033023] Lustre: Skipped 1 previous similar message [ 4402.146349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4402.150560] Lustre: Skipped 2 previous similar messages [ 4402.456443] Lustre: server umount lustre-OST0000 complete [ 4403.957690] LustreError: 168193:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4403.960260] Lustre: 167542:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4403.964233] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4403.973120] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4403.979065] LustreError: 167542:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008b4c14ef x1876223918640640/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4403.987687] LustreError: 167542:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4403.991892] LustreError: 167542:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4403.997063] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 4404.119805] Lustre: server umount lustre-MDT0000 complete [ 4405.729596] LustreError: 164445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306632 with bad export cookie 5847874358942602487 [ 4405.736340] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4405.736598] LustreError: 163775:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4411.903248] Lustre: server umount lustre-MDT0001 complete [ 4416.123225] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4417.279502] Key type lgssc unregistered [ 4417.429695] LNet: 168944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4417.433301] LNetError: 168944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4417.443353] LNet: Removed LNI 192.168.204.111@tcp [ 4417.794202] Key type .llcrypt unregistered [ 4417.795723] Key type ._llcrypt unregistered [ 4428.353138] Key type ._llcrypt registered [ 4428.355247] Key type .llcrypt registered [ 4428.397514] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4428.768767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4428.804050] alg: No test for adler32 (adler32-zlib) [ 4429.691062] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 4429.797924] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4431.391198] Key type lgssc registered [ 4431.914761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4435.467717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4435.478902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.684195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4438.455380] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4441.865966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4442.042289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4443.584325] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4446.503536] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4448.630583] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4451.422474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4451.574240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4453.784548] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4456.562622] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4461.801765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4467.170348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4467.171437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4467.172150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4467.177391] Lustre: Skipped 1 previous similar message [ 4470.432676] Lustre: server umount lustre-OST0000 complete [ 4472.290372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4472.292466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4472.297354] Lustre: Skipped 1 previous similar message [ 4472.301541] Lustre: Skipped 1 previous similar message [ 4477.408457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4477.412994] Lustre: Skipped 2 previous similar messages [ 4478.135982] Lustre: server umount lustre-MDT0000 complete [ 4479.608214] LustreError: 170311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306706 with bad export cookie 10029734942440058662 [ 4479.611235] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4479.612475] LustreError: 170311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4479.726716] Lustre: server umount lustre-MDT0001 complete [ 4483.846027] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4485.030518] Key type lgssc unregistered [ 4485.170522] LNet: 173855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4485.173920] LNetError: 173855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4485.182329] LNet: Removed LNI 192.168.204.111@tcp [ 4485.520052] Key type .llcrypt unregistered [ 4485.521757] Key type ._llcrypt unregistered [ 4499.222785] Key type ._llcrypt registered [ 4499.224464] Key type .llcrypt registered [ 4499.271886] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4499.722360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4499.733498] alg: No test for adler32 (adler32-zlib) [ 4500.597892] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 4500.695393] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4502.279211] Key type lgssc registered [ 4502.693952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4505.577533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4505.585418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.758070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4508.344763] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4511.650985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4511.799505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4513.377995] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4516.038143] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4518.256254] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4521.178527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4521.336712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4523.571521] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4526.265313] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4531.558093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4541.923302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4541.924247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4541.926848] LustreError: Skipped 1 previous similar message [ 4541.928693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4541.945825] Lustre: Skipped 1 previous similar message [ 4545.694433] Lustre: server umount lustre-OST0000 complete [ 4547.552890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4547.556413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4547.561098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4547.563594] Lustre: Skipped 1 previous similar message [ 4552.161088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4552.161637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4553.371069] Lustre: server umount lustre-MDT0000 complete [ 4554.744635] LustreError: 177367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306781 with bad export cookie 12907167252290119095 [ 4554.748728] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4554.750718] LustreError: 177367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4554.887120] Lustre: server umount lustre-MDT0001 complete [ 4558.535676] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4559.624510] Key type lgssc unregistered [ 4559.766420] LNet: 179075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4559.769971] LNetError: 179075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4559.777327] LNet: Removed LNI 192.168.204.111@tcp [ 4560.121209] Key type .llcrypt unregistered [ 4560.122155] Key type ._llcrypt unregistered [ 4568.702182] Key type ._llcrypt registered [ 4568.703518] Key type .llcrypt registered [ 4568.739413] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4569.058614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4569.106089] alg: No test for adler32 (adler32-zlib) [ 4569.964409] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 4570.056278] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4571.639179] Key type lgssc registered [ 4572.031697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4575.918320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4575.924150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4577.059777] Lustre: *** cfs_fail_loc=135, val=0*** [ 4577.104834] LustreError: 180570:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 4577.107420] LustreError: 180570:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.111@tcp: cfg command failed: rc = -2 [ 4577.110863] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4577.110863] [ 4577.114928] LustreError: MGC192.168.204.111@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4577.119991] LustreError: 180532:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4577.123486] LustreError: 180532:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 4577.126794] LustreError: 180532:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 4577.128790] LustreError: 180532:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 4577.161480] Lustre: server umount lustre-MDT0000 complete [ 4577.163182] LustreError: 180532:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4579.667274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4579.830077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4581.312073] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4584.310755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.447669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4585.947818] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4588.567369] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4590.570809] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4594.336856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4594.337772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4594.341678] Lustre: Skipped 1 previous similar message [ 4597.393534] Lustre: server umount lustre-MDT0000 complete [ 4598.701104] LustreError: 180966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306825 with bad export cookie 1412261834086404856 [ 4598.704095] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4598.705728] LustreError: 180966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4598.833270] Lustre: server umount lustre-MDT0001 complete [ 4602.814121] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4603.944067] Key type lgssc unregistered [ 4604.109494] LNet: 183256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4604.115344] LNetError: 183256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4604.126362] LNet: Removed LNI 192.168.204.111@tcp [ 4604.466506] Key type .llcrypt unregistered [ 4604.468388] Key type ._llcrypt unregistered [ 4612.789811] Key type ._llcrypt registered [ 4612.791410] Key type .llcrypt registered [ 4612.836848] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4612.960379] Key type .llcrypt unregistered [ 4612.961772] Key type ._llcrypt unregistered [ 4620.268826] Key type ._llcrypt registered [ 4620.270157] Key type .llcrypt registered [ 4620.314871] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4620.663397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4620.721481] alg: No test for adler32 (adler32-zlib) [ 4621.586888] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 4621.697512] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4623.287219] Key type lgssc registered [ 4623.692532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4626.839197] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4626.852586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4642.335516] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4647.455317] LustreError: 185356:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 4674.079186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.055567] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4678.569030] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4680.830313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4680.997225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4681.016890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4682.347293] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4685.293164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4685.447055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4686.882928] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4689.563691] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4691.602549] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4692.718639] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4699.104644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4699.106426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4699.110048] Lustre: Skipped 1 previous similar message [ 4704.227162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4704.230342] Lustre: Skipped 2 previous similar messages [ 4705.173222] Lustre: server umount lustre-OST0000 complete [ 4709.344633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4709.345572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4709.348639] Lustre: Skipped 1 previous similar message [ 4712.742836] Lustre: server umount lustre-MDT0000 complete [ 4714.197617] LustreError: 188019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306941 with bad export cookie 4223245405970553714 [ 4714.201039] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4714.201829] LustreError: 188019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4714.313231] Lustre: server umount lustre-MDT0001 complete [ 4718.300484] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4719.409794] Key type lgssc unregistered [ 4719.551598] LNet: 189320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4719.554765] LNetError: 189320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4719.564468] LNet: Removed LNI 192.168.204.111@tcp [ 4719.893510] Key type .llcrypt unregistered [ 4719.894963] Key type ._llcrypt unregistered [ 4729.489467] Key type ._llcrypt registered [ 4729.491368] Key type .llcrypt registered [ 4729.538558] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4729.988444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4730.039400] alg: No test for adler32 (adler32-zlib) [ 4730.900330] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 4730.999727] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4732.591667] Key type lgssc registered [ 4732.990737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4735.769923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4735.776507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4736.937103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4738.390680] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4741.415765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4741.563126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4743.010435] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4745.739497] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4747.778693] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4750.254060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4750.377505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.376755] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4754.907905] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4762.660238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4766.503979] Lustre: Failing over lustre-MDT0000 [ 4766.596293] Lustre: server umount lustre-MDT0000 complete [ 4767.199958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4767.203259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4767.208438] LustreError: 190705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4767.213317] LustreError: 190705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4767.713995] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4767.715212] LustreError: 192707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4767.718809] Lustre: Skipped 1 previous similar message [ 4770.393447] LustreError: 192089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.399944] LustreError: 192089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4772.832091] LustreError: 190704:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4772.838681] LustreError: 190704:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4777.954809] LustreError: 190704:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4777.962905] LustreError: 190704:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4779.867921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4779.919604] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4780.035485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4780.066843] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4780.626324] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4781.516471] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4783.222555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4785.127516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4785.135424] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4785.154117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4787.906102] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4795.361609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4795.366717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4795.367936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4795.372885] Lustre: Skipped 1 previous similar message [ 4795.377029] Lustre: Skipped 1 previous similar message [ 4800.471275] Lustre: server umount lustre-OST0000 complete [ 4805.600925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4805.600939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4805.609163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4805.613897] Lustre: Skipped 1 previous similar message [ 4810.720279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4810.723084] Lustre: Skipped 1 previous similar message [ 4815.840621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4815.843238] Lustre: Skipped 1 previous similar message [ 4816.351156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4816.415513] Lustre: server umount lustre-MDT0000 complete [ 4817.884242] LustreError: 190685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307045 with bad export cookie 4399194681021229477 [ 4817.886661] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4817.888059] LustreError: 190685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4818.009844] Lustre: server umount lustre-MDT0001 complete [ 4822.161766] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4823.384241] Key type lgssc unregistered [ 4823.526475] LNet: 195604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4823.529827] LNetError: 195604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4823.538385] LNet: Removed LNI 192.168.204.111@tcp [ 4823.938646] Key type .llcrypt unregistered [ 4823.939906] Key type ._llcrypt unregistered [ 4834.798187] Key type ._llcrypt registered [ 4834.799305] Key type .llcrypt registered [ 4834.849254] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4835.318878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4835.370963] alg: No test for adler32 (adler32-zlib) [ 4836.271348] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 4836.376182] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4837.975501] Key type lgssc registered [ 4838.505633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4842.039612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4842.049602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4843.262387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4844.968083] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4848.859985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4849.048324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4850.737332] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4853.623839] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4855.830695] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4858.511127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4858.650993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4860.924377] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4863.808337] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4867.881037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4869.726592] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4872.259168] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4883.215688] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4890.591941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4890.595374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4890.600550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4893.665452] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4893.666598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.666718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4895.698626] Lustre: server umount lustre-OST0000 complete [ 4898.785942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4898.788627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4898.792026] Lustre: Skipped 1 previous similar message [ 4898.794809] Lustre: Skipped 1 previous similar message [ 4903.258124] Lustre: server umount lustre-MDT0000 complete [ 4903.903953] LustreError: 197715:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4903.909091] LustreError: 197715:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4904.648163] LustreError: 196969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307131 with bad export cookie 13763941246594907232 [ 4904.651718] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4904.653806] LustreError: 196969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4904.780577] Lustre: server umount lustre-MDT0001 complete [ 4909.023163] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4910.243826] Key type lgssc unregistered [ 4910.386502] LNet: 200827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4910.390090] LNetError: 200827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4910.400475] LNet: Removed LNI 192.168.204.111@tcp [ 4910.784282] Key type .llcrypt unregistered [ 4910.785743] Key type ._llcrypt unregistered [ 4919.041611] Key type ._llcrypt registered [ 4919.042853] Key type .llcrypt registered [ 4919.086778] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4919.448484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4919.490246] alg: No test for adler32 (adler32-zlib) [ 4920.357400] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 4920.461592] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4922.047216] Key type lgssc registered [ 4922.414456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4925.574724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4925.581779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4926.757962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4928.337858] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4931.578648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4931.720329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4933.318077] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4936.214787] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4938.369161] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4941.010130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4941.160423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4943.434167] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4946.250900] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4952.425963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4959.255279] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4968.457472] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4978.149941] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4984.777115] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4997.602364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4997.605367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4997.611199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4997.613823] Lustre: Skipped 1 previous similar message [ 5002.720976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5002.722985] Lustre: Skipped 1 previous similar message [ 5007.841401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5007.843185] Lustre: Skipped 1 previous similar message [ 5008.351137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5008.410790] Lustre: server umount lustre-OST0000 complete [ 5012.960646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5012.964353] Lustre: Skipped 1 previous similar message [ 5012.966639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5012.968713] Lustre: Skipped 1 previous similar message [ 5018.080458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5018.083771] Lustre: Skipped 1 previous similar message [ 5024.223120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5024.289889] Lustre: server umount lustre-MDT0000 complete [ 5025.828784] LustreError: 202045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307252 with bad export cookie 3051032926785809472 [ 5025.829699] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5025.833532] LustreError: 202045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5025.985461] Lustre: server umount lustre-MDT0001 complete [ 5030.088930] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5031.259434] Key type lgssc unregistered [ 5031.402483] LNet: 207591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5031.404994] LNetError: 207591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5031.413463] LNet: Removed LNI 192.168.204.111@tcp [ 5031.749809] Key type .llcrypt unregistered [ 5031.751075] Key type ._llcrypt unregistered [ 5041.245131] Key type ._llcrypt registered [ 5041.246462] Key type .llcrypt registered [ 5041.288838] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5041.617300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5041.647570] alg: No test for adler32 (adler32-zlib) [ 5042.510527] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 5042.603820] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5044.191177] Key type lgssc registered [ 5044.577620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5047.335186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5047.342037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5048.510907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5049.944174] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5053.265857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5053.399789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5054.969944] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5057.751292] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5059.804651] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5062.234471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5062.372791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.477521] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5067.164503] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5073.638833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5084.131149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5084.134937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5084.135294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5084.136346] Lustre: Skipped 1 previous similar message [ 5087.698833] Lustre: server umount lustre-OST0000 complete [ 5089.248324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5089.248433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.248994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5089.249016] Lustre: Skipped 1 previous similar message [ 5089.250375] LustreError: Skipped 1 previous similar message [ 5089.253749] Lustre: Skipped 1 previous similar message [ 5094.368269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5094.370856] Lustre: Skipped 2 previous similar messages [ 5095.189431] Lustre: server umount lustre-MDT0000 complete [ 5096.565409] LustreError: 211638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307323 with bad export cookie 15241924967862430370 [ 5096.568599] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5096.570765] LustreError: 211638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5096.702778] Lustre: server umount lustre-MDT0001 complete [ 5100.681974] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5101.843962] Key type lgssc unregistered [ 5101.995869] LNet: 212675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5101.999539] LNetError: 212675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5102.009534] LNet: Removed LNI 192.168.204.111@tcp [ 5102.360090] Key type .llcrypt unregistered [ 5102.361617] Key type ._llcrypt unregistered [ 5110.228799] Key type ._llcrypt registered [ 5110.230100] Key type .llcrypt registered [ 5110.266137] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5110.648929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5110.677049] alg: No test for adler32 (adler32-zlib) [ 5111.527777] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 5111.613725] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5113.199227] Key type lgssc registered [ 5113.673799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5116.780845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5116.792518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5118.000699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5119.541927] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5122.593376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.719426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5124.345351] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5126.967959] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5129.024829] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5131.757928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5131.945092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5134.275513] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5137.166598] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5144.232338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5153.760902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5153.763894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5153.768943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5154.787284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5154.787443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5154.787594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5154.797017] Lustre: Skipped 1 previous similar message [ 5158.418614] Lustre: server umount lustre-OST0000 complete [ 5159.904292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5159.904960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5159.908100] Lustre: Skipped 1 previous similar message [ 5165.024426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5165.026587] Lustre: Skipped 2 previous similar messages [ 5170.144087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5170.146166] Lustre: Skipped 1 previous similar message [ 5174.239114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5174.300685] Lustre: server umount lustre-MDT0000 complete [ 5175.264213] LustreError: 214651:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5175.271218] LustreError: 214651:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5175.542356] LustreError: 213894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307402 with bad export cookie 1296886859014547059 [ 5175.545287] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5175.545713] LustreError: 213894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5175.660645] Lustre: server umount lustre-MDT0001 complete [ 5179.275933] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5180.365726] Key type lgssc unregistered [ 5180.519506] LNet: 217612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5180.522447] LNetError: 217612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5180.534325] LNet: Removed LNI 192.168.204.111@tcp [ 5180.848154] Key type .llcrypt unregistered [ 5180.849647] Key type ._llcrypt unregistered [ 5188.459491] Key type ._llcrypt registered [ 5188.461503] Key type .llcrypt registered [ 5188.498839] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5188.838752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5188.898225] alg: No test for adler32 (adler32-zlib) [ 5189.775350] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 5189.874618] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5191.471209] Key type lgssc registered [ 5191.888949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5194.757525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5194.763684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5195.915093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5197.396959] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5200.597402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.753640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5202.294449] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5204.937628] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5206.803908] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5209.217468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5209.352907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5211.444228] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5214.259864] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5222.694955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5227.673271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5227.699165] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5227.769146] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5229.884374] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5232.338950] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5232.840721] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5243.194188] Lustre: Permanently deactivating lustre-OST0001 [ 5243.199710] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5253.630165] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5255.058163] Lustre: Permanently reactivating lustre-OST0001 [ 5255.063600] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5255.065796] Lustre: Skipped 2 previous similar messages [ 5261.815319] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5261.821292] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5261.824680] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5261.828920] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5261.840890] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5266.912156] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5266.914740] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5266.918217] Lustre: Skipped 1 previous similar message [ 5266.920275] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5266.921709] Lustre: Skipped 1 previous similar message [ 5268.963510] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5268.965309] Lustre: Skipped 1 previous similar message [ 5272.504900] Lustre: server umount lustre-OST0001 complete [ 5274.083335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5274.085037] LustreError: 220671:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5274.085475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5274.085482] Lustre: Skipped 1 previous similar message [ 5274.086325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5274.086363] LustreError: Skipped 1 previous similar message [ 5274.090707] LustreError: 220671:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5279.201545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5279.202428] LustreError: 220669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5279.203617] Lustre: Skipped 2 previous similar messages [ 5279.208118] LustreError: 220669:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5279.887278] Lustre: server umount lustre-OST0000 complete [ 5282.784924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5282.787425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5282.793081] Lustre: Skipped 1 previous similar message [ 5284.320238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5284.322718] Lustre: Skipped 2 previous similar messages [ 5287.318538] Lustre: server umount lustre-MDT0000 complete [ 5288.462982] LustreError: 221711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307515 with bad export cookie 13695717146982187132 [ 5288.466546] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5288.466905] LustreError: 221711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5288.586193] Lustre: server umount lustre-MDT0001 complete [ 5291.882298] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5292.880697] Key type lgssc unregistered [ 5292.997336] LNet: 223934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5292.999246] LNetError: 223934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5293.007356] LNet: Removed LNI 192.168.204.111@tcp [ 5293.280812] Key type .llcrypt unregistered [ 5293.281817] Key type ._llcrypt unregistered [ 5307.133383] Key type ._llcrypt registered [ 5307.134804] Key type .llcrypt registered [ 5307.175785] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5307.609115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5307.616945] alg: No test for adler32 (adler32-zlib) [ 5308.482711] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 5308.588437] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5310.167271] Key type lgssc registered [ 5310.540472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5313.185577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5313.191629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5314.280464] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5314.289668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5314.346895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5315.674950] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5318.475809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5318.515226] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5318.526296] Lustre: 226111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5318.618741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5319.954895] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5322.384172] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5324.271985] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5326.561939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5326.629772] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5326.692480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5328.556894] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5330.768809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 5330.948361] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5332.504850] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5336.982306] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5345.511443] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5355.088913] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5363.651516] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5379.758450] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5379.761471] Lustre: Skipped 1 previous similar message [ 5414.350748] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5414.353039] Lustre: Skipped 4 previous similar messages [ 5478.914444] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5478.917372] Lustre: Skipped 7 previous similar messages [ 5504.994016] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5504.995741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5504.996655] LustreError: Skipped 1 previous similar message [ 5505.000966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5505.004335] Lustre: Skipped 1 previous similar message [ 5506.256773] Lustre: server umount lustre-OST0000 complete [ 5508.063900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5508.066559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.070748] Lustre: Skipped 1 previous similar message [ 5508.072671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5510.112426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5510.112739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5513.625857] Lustre: server umount lustre-MDT0000 complete [ 5514.945295] LustreError: 228676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307742 with bad export cookie 15294014012498498737 [ 5514.948080] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5514.949748] LustreError: 228676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5515.072980] Lustre: server umount lustre-MDT0001 complete [ 5518.850127] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5519.929270] Key type lgssc unregistered [ 5520.065362] LNet: 230914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5520.068760] LNetError: 230914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5520.079646] LNet: Removed LNI 192.168.204.111@tcp [ 5520.405751] Key type .llcrypt unregistered [ 5520.406973] Key type ._llcrypt unregistered [ 5530.104908] Key type ._llcrypt registered [ 5530.106049] Key type .llcrypt registered [ 5530.145947] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5530.467371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5530.474820] alg: No test for adler32 (adler32-zlib) [ 5531.334371] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 5531.439276] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5533.031337] Key type lgssc registered [ 5533.448508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5536.289638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5536.296071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5537.453784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5538.827968] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5541.698311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.831779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5543.176379] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5545.416297] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5547.186289] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5549.363390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5549.479835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5551.355664] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5553.635710] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5562.788477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 5573.089064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.090764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5573.093142] Lustre: Skipped 1 previous similar message [ 5573.098356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5574.114015] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5574.116247] Lustre: Skipped 2 previous similar messages [ 5578.895049] Lustre: server umount lustre-OST0000 complete [ 5580.154629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.155569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5580.157884] Lustre: Skipped 1 previous similar message [ 5586.330153] Lustre: server umount lustre-MDT0000 complete [ 5586.399823] LustreError: 232310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5586.403612] LustreError: 232310:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5587.622115] LustreError: 232334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307814 with bad export cookie 9423002961737269015 [ 5587.624349] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5587.625580] LustreError: 232334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5587.725306] Lustre: server umount lustre-MDT0001 complete [ 5591.210947] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5592.174660] Key type lgssc unregistered [ 5592.304382] LNet: 236018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5592.307266] LNetError: 236018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5592.313295] LNet: Removed LNI 192.168.204.111@tcp [ 5592.615489] Key type .llcrypt unregistered [ 5592.616652] Key type ._llcrypt unregistered [ 5600.900584] Key type ._llcrypt registered [ 5600.901600] Key type .llcrypt registered [ 5600.936925] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5601.337781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5601.354027] alg: No test for adler32 (adler32-zlib) [ 5602.209593] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 5602.309438] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5603.887185] Key type lgssc registered [ 5604.291599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5869.376060] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5870.311476] Key type lgssc unregistered [ 5870.425248] LNet: 238027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5870.427358] LNetError: 238027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5870.433242] LNet: Removed LNI 192.168.204.111@tcp [ 5870.708451] Key type .llcrypt unregistered [ 5870.709922] Key type ._llcrypt unregistered [ 5878.525226] Key type ._llcrypt registered [ 5878.526611] Key type .llcrypt registered [ 5878.569294] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 5884.869382] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5885.202666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5885.230047] alg: No test for adler32 (adler32-zlib) [ 5886.085947] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 5886.175092] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5887.759140] Key type lgssc registered [ 5888.106283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5898.409568] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5617 sec ========== 10:02:05 (1789308125) [ 5899.013675] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:02:05 (1789308125) === [ 5900.231050] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:02:06 (1789308126) === [ 5914.325975] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5915.407612] Key type lgssc unregistered [ 5915.536383] LNet: 243241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5915.539714] LNetError: 243241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5915.548422] LNet: Removed LNI 192.168.204.111@tcp [ 5915.871432] Key type .llcrypt unregistered [ 5915.873249] Key type ._llcrypt unregistered