[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 500719984 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002261] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004021] kvm-guest: setup PV IPIs [ 0.007591] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008044] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010176] LSM: Security Framework initializing [ 0.011103] Yama: becoming mindful. [ 0.012082] SELinux: Initializing. [ 0.013070] *** VALIDATE selinux *** [ 0.021027] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025581] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028124] *** VALIDATE tmpfs *** [ 0.029499] *** VALIDATE proc *** [ 0.030247] *** VALIDATE cgroup *** [ 0.031006] *** VALIDATE cgroup2 *** [ 0.033052] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034126] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036030] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.039890] debug: unmapping init [mem 0xffffffffa9a59000-0xffffffffa9a60fff] [ 0.041148] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042544] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043020] ... version: 2 [ 0.044007] ... bit width: 48 [ 0.044943] ... generic registers: 4 [ 0.045010] ... value mask: 0000ffffffffffff [ 0.046015] ... max period: 00007fffffffffff [ 0.047015] ... fixed-purpose events: 3 [ 0.048009] ... event mask: 000000070000000f [ 0.050219] rcu: Hierarchical SRCU implementation. [ 0.053043] smp: Bringing up secondary CPUs ... [ 0.054776] x86: Booting SMP configuration: [ 0.055034] .... node #0, CPUs: #1 #2 #3 [ 0.062080] smp: Brought up 1 node, 4 CPUs [ 0.064011] smpboot: Max logical packages: 1 [ 0.065015] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.128079] node 0 deferred pages initialised in 60ms [ 0.131231] devtmpfs: initialized [ 0.132276] x86/mm: Memory block size: 128MB [ 0.134807] gcov: version magic: 0x41383552 [ 0.136035] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.140455] pinctrl core: initialized pinctrl subsystem [ 0.141168] [ 0.143020] ************************************************************* [ 0.148026] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153023] ** ** [ 0.157023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160016] ** ** [ 0.165022] ** This means that this kernel is built to expose internal ** [ 0.169030] ** IOMMU data structures, which may compromise security on ** [ 0.174137] ** your system. ** [ 0.178105] ** ** [ 0.182025] ** If you see this message and you are not debugging the ** [ 0.185020] ** kernel, report this immediately to your vendor! ** [ 0.187019] ** ** [ 0.191023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198020] ************************************************************* [ 0.204000] NET: Registered protocol family 16 [ 0.207672] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.210076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.213088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.216033] cpuidle: using governor menu [ 0.217932] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.218476] PCI: Using configuration type 1 for base access [ 0.219113] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.226337] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.231145] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.236186] cryptd: max_cpu_qlen set to 1000 [ 0.241306] ACPI: Added _OSI(Module Device) [ 0.243016] ACPI: Added _OSI(Processor Device) [ 0.244013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.246014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.251000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.258585] ACPI: Interpreter enabled [ 0.261079] ACPI: PM: (supports S0 S3 S4 S5) [ 0.262013] ACPI: Using IOAPIC for interrupt routing [ 0.264117] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.268768] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.284751] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.287038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.291018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.297870] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.311383] acpiphp: Slot [2] registered [ 0.314332] acpiphp: Slot [5] registered [ 0.318641] acpiphp: Slot [6] registered [ 0.322318] acpiphp: Slot [7] registered [ 0.325167] acpiphp: Slot [8] registered [ 0.328184] acpiphp: Slot [9] registered [ 0.330262] acpiphp: Slot [10] registered [ 0.333132] acpiphp: Slot [3] registered [ 0.336159] acpiphp: Slot [4] registered [ 0.339112] acpiphp: Slot [11] registered [ 0.341174] acpiphp: Slot [12] registered [ 0.344209] acpiphp: Slot [13] registered [ 0.346284] acpiphp: Slot [14] registered [ 0.349142] acpiphp: Slot [15] registered [ 0.351149] acpiphp: Slot [16] registered [ 0.353216] acpiphp: Slot [17] registered [ 0.355224] acpiphp: Slot [18] registered [ 0.358182] acpiphp: Slot [19] registered [ 0.360151] acpiphp: Slot [20] registered [ 0.363216] acpiphp: Slot [21] registered [ 0.366188] acpiphp: Slot [22] registered [ 0.368122] acpiphp: Slot [23] registered [ 0.371103] acpiphp: Slot [24] registered [ 0.373116] acpiphp: Slot [25] registered [ 0.376192] acpiphp: Slot [26] registered [ 0.378301] acpiphp: Slot [27] registered [ 0.381117] acpiphp: Slot [28] registered [ 0.384147] acpiphp: Slot [29] registered [ 0.387200] acpiphp: Slot [30] registered [ 0.391119] acpiphp: Slot [31] registered [ 0.393059] PCI host bridge to bus 0000:00 [ 0.395014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.400022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.402049] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.405032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.409032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.412023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.413000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.413000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.415000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.423013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.430990] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.435027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.441019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.445016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.449438] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.455601] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.460058] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.464342] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.472019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.493018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.500019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.505325] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.513010] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.525015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.550019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.566000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.580015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.598015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.628486] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.643822] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.650119] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.657020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.677015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.685426] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.691000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.691000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.708019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.726446] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.740014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.743012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.769022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.799654] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.806019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.814033] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.838023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.839000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.839000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.839000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.843441] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.848558] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.856306] iommu: Default domain type: Passthrough [ 0.865630] SCSI subsystem initialized [ 0.869473] ACPI: bus type USB registered [ 0.872233] usbcore: registered new interface driver usbfs [ 0.876320] usbcore: registered new interface driver hub [ 0.879299] usbcore: registered new device driver usb [ 0.882210] pps_core: LinuxPPS API ver. 1 registered [ 0.887019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.896087] PTP clock support registered [ 0.901149] EDAC MC: Ver: 3.0.0 [ 0.903476] PCI: Using ACPI for IRQ routing [ 0.905649] NetLabel: Initializing [ 0.907014] NetLabel: domain hash size = 128 [ 0.909010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.910100] NetLabel: unlabeled traffic allowed by default [ 0.912378] vgaarb: loaded [ 0.914397] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.916014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.922130] clocksource: Switched to clocksource kvm-clock [ 1.050920] VFS: Disk quotas dquot_6.6.0 [ 1.052897] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.055952] *** VALIDATE ramfs *** [ 1.057623] *** VALIDATE hugetlbfs *** [ 1.059540] pnp: PnP ACPI init [ 1.062514] pnp: PnP ACPI: found 6 devices [ 1.080212] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.084225] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.086574] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.088874] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.091706] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.094641] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.097712] NET: Registered protocol family 2 [ 1.099927] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.105472] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.109484] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.115375] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.118986] TCP: Hash tables configured (established 65536 bind 65536) [ 1.122161] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.126327] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.129771] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.133494] NET: Registered protocol family 1 [ 1.137456] RPC: Registered named UNIX socket transport module. [ 1.140637] RPC: Registered udp transport module. [ 1.142118] RPC: Registered tcp transport module. [ 1.143881] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.146714] NET: Registered protocol family 44 [ 1.148498] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.150387] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.152301] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.155103] PCI: CLS 0 bytes, default 64 [ 1.157785] Unpacking initramfs... [ 3.159805] debug: unmapping init [mem 0xffff8be87cc54000-0xffff8be87ffbffff] [ 3.166573] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.168639] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.172454] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 4.058980] Initialise system trusted keyrings [ 4.061412] Key type blacklist registered [ 4.066771] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.103499] zbud: loaded [ 4.122791] *** VALIDATE nfs *** [ 4.124018] *** VALIDATE nfs4 *** [ 4.125312] pstore: using deflate compression [ 4.129359] Platform Keyring initialized [ 4.237764] NET: Registered protocol family 38 [ 4.239868] Key type asymmetric registered [ 4.241940] Asymmetric key parser 'x509' registered [ 4.243963] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.248297] io scheduler mq-deadline registered [ 4.249851] io scheduler kyber registered [ 4.251893] io scheduler bfq registered [ 4.254105] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.256980] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.259542] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.262273] ACPI: Power Button [PWRF] [ 4.267980] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.274346] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.288231] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.294366] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.309981] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.343280] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.374247] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.378720] Non-volatile memory driver v1.3 [ 4.380279] Linux agpgart interface v0.103 [ 4.411545] virtio_blk virtio1: [vda] 145256 512-byte logical blocks (74.4 MB/70.9 MiB) [ 4.415422] vda: detected capacity change from 0 to 74371072 [ 4.435971] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.439031] vdb: detected capacity change from 0 to 1073741824 [ 4.454093] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.458044] vdc: detected capacity change from 0 to 2621440000 [ 4.476918] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.480475] vdd: detected capacity change from 0 to 2621440000 [ 4.497260] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.500077] vde: detected capacity change from 0 to 4294967296 [ 4.515550] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.518714] vdf: detected capacity change from 0 to 4294967296 [ 4.528985] libphy: Fixed MDIO Bus: probed [ 4.535758] usbcore: registered new interface driver usbserial_generic [ 4.539397] usbserial: USB Serial support registered for generic [ 4.543553] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.549579] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.552290] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.555288] mousedev: PS/2 mouse device common for all mice [ 4.558920] rtc_cmos 00:05: RTC can wake from S4 [ 4.564716] rtc_cmos 00:05: registered as rtc0 [ 4.566953] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.567832] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.575077] intel_pstate: CPU model not supported [ 4.580663] hid: raw HID events driver (C) Jiri Kosina [ 4.586795] usbcore: registered new interface driver usbhid [ 4.589689] usbhid: USB HID core driver [ 4.591710] drop_monitor: Initializing network drop monitor service [ 4.595304] Initializing XFRM netlink socket [ 4.597503] NET: Registered protocol family 10 [ 4.602241] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.605990] Segment Routing with IPv6 [ 4.612092] NET: Registered protocol family 17 [ 4.612421] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.614710] mpls_gso: MPLS GSO support [ 4.625934] RAS: Correctable Errors collector initialized. [ 4.628630] AVX version of gcm_enc/dec engaged. [ 4.630820] AES CTR mode by8 optimization enabled [ 4.710954] sched_clock: Marking stable (4710884804, 0)->(5909762487, -1198877683) [ 4.714309] registered taskstats version 1 [ 4.715889] Loading compiled-in X.509 certificates [ 4.717728] zswap: loaded using pool lzo/zbud [ 4.817809] Key type big_key registered [ 4.829658] Key type encrypted registered [ 4.831107] ima: No TPM chip found, activating TPM-bypass! [ 4.832887] ima: Allocated hash algorithm: sha1 [ 4.834520] ima: No architecture policies found [ 4.835907] evm: Initialising EVM extended attributes: [ 4.837511] evm: security.selinux [ 4.838553] evm: security.ima [ 4.839575] evm: security.capability [ 4.840655] evm: HMAC attrs: 0x1 [ 4.842791] rtc_cmos 00:05: setting system clock to 2026-07-09 01:55:42 UTC (1783562142) [ 4.849741] debug: unmapping init [mem 0xffffffffaaa03000-0xffffffffaabfffff] [ 4.854440] debug: unmapping init [mem 0xffffffffa9782000-0xffffffffa9a58fff] [ 4.867255] Write protecting the kernel read-only data: 28672k [ 4.872447] debug: unmapping init [mem 0xffffffffa7e03000-0xffffffffa7ffffff] [ 4.876378] debug: unmapping init [mem 0xffffffffa8714000-0xffffffffa87fffff] [ 4.924729] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.938628] systemd[1]: Detected virtualization kvm. [ 4.941436] systemd[1]: Detected architecture x86-64. [ 4.944306] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.978443] systemd[1]: No hostname configured. [ 4.981461] systemd[1]: Set hostname to . [ 4.984683] random: systemd: uninitialized urandom read (16 bytes read) [ 4.988825] systemd[1]: Initializing machine ID from random generator. [ 5.427770] random: systemd: uninitialized urandom read (16 bytes read) [ 5.432212] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.443488] random: systemd: uninitialized urandom read (16 bytes read) [ 5.446724] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.454362] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.920717] device-mapper: uevent: version 1.0.3 [ 6.924960] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 8.144540] random: fast init done [ 8.170119] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.771042] scsi host0: ata_piix [ 8.779738] scsi host1: ata_piix [ 8.782401] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.789157] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.572369] random: crng init done [ 13.581547] random: 7 urandom warning(s) missed due to ratelimiting [ 15.940327] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.768945] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.780897] printk: systemd: 25 output lines suppressed due to ratelimiting [ 20.460667] SELinux: Disabled at runtime. [ 20.552592] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.562673] systemd[1]: Detected virtualization kvm. [ 20.565437] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.687201] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.691513] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.701301] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.707620] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.712768] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.723939] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.742866] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ 22.309543] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 23.296529] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.343727] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.412420] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.019130] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.084722] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 29.744576] Key type dns_resolver registered [ 30.296831] NFS: Registering the id_resolver key type [ 30.299681] Key type id_resolver registered [ 30.302114] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg336-server login: [ 47.793150] hrtimer: interrupt took 2246344 ns [ 65.859339] libcfs: loading out-of-tree module taints kernel. [ 65.910302] Key type ._llcrypt registered [ 65.914476] Key type .llcrypt registered [ 66.006889] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 77.491216] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 78.406293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 78.418889] alg: No test for adler32 (adler32-zlib) [ 79.615724] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 80.174329] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 81.871325] Key type lgssc registered [ 82.769174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.167745] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 110.424680] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 116.818542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 116.830156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 117.957184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 117.978232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 118.029728] Lustre: lustre-MDT0000: new disk, initializing [ 118.076763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 118.088393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 120.174773] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 127.259663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 127.318872] Lustre: 6482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 127.343854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 127.346878] Lustre: Skipped 1 previous similar message [ 127.399238] Lustre: lustre-MDT0001: new disk, initializing [ 127.431649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 127.450393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 127.458238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 129.579940] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 132.425096] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 137.313611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 137.445183] Lustre: lustre-OST0000: new disk, initializing [ 137.448729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 137.454699] Lustre: 8420:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 137.496844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 140.745103] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 141.372586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 141.379982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 141.405812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 148.835086] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 148.916080] Lustre: lustre-OST0001: new disk, initializing [ 148.919037] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 148.925267] Lustre: 9490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 148.960273] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 152.235296] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.706595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 158.711319] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 158.739851] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 160.435587] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 165.775587] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 168.779103] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing check_logdir /tmp/testlogs/ [ 171.040549] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing yml_node [ 173.073245] Lustre: DEBUG MARKER: Client: 2.17.54.105 [ 174.263708] Lustre: DEBUG MARKER: MDS: 2.17.54.105 [ 175.400349] Lustre: DEBUG MARKER: OSS: 2.17.54.105 [ 176.193178] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jul 8 21:58:32 EDT 2026 [ 184.072310] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 184.739332] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 199.135697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 199.139316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 199.145477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 199.647767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 199.648514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 199.654254] Lustre: Skipped 1 previous similar message [ 203.369793] Lustre: server umount lustre-MDT0000 complete [ 204.833837] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562342 with bad export cookie 14366380399670481493 [ 204.835663] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 204.839069] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 209.888063] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 209.888243] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 209.893496] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 209.893515] Lustre: Skipped 2 previous similar messages [ 209.896172] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 209.902458] Lustre: Skipped 2 previous similar messages [ 215.008029] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 215.008053] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 215.017607] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 215.021100] Lustre: Skipped 2 previous similar messages [ 216.032069] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 216.037910] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 219.103148] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 219.183283] Lustre: server umount lustre-MDT0001 complete [ 220.699839] Lustre: server umount lustre-OST0000 complete [ 222.267656] Lustre: server umount lustre-OST0001 complete [ 224.733445] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 228.271586] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 242.498023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 242.596297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 242.610919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 242.651401] Lustre: lustre-MDT0000: new disk, initializing [ 242.679458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.688442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 244.275530] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 249.018050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 249.071198] Lustre: 15031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 249.194372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 250.762504] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 254.876534] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 257.117135] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 259.956474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.090518] Lustre: lustre-OST0000: new disk, initializing [ 260.093268] Lustre: Skipped 1 previous similar message [ 260.095767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 260.099178] Lustre: Skipped 2 previous similar messages [ 260.102742] Lustre: 17002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.141659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 260.147815] Lustre: Skipped 1 previous similar message [ 261.768677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 261.773016] Lustre: Skipped 1 previous similar message [ 261.775257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 261.820579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 262.429096] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 266.573880] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 268.156997] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 268.238454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 269.814526] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 269.899902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 272.352903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 272.355949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 272.356638] LustreError: Skipped 1 previous similar message [ 272.363874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 272.367857] Lustre: Skipped 3 previous similar messages [ 276.755576] Lustre: server umount lustre-OST0000 complete [ 280.032419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 280.035713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.042090] Lustre: Skipped 1 previous similar message [ 282.592213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 282.595265] Lustre: Skipped 2 previous similar messages [ 284.320803] Lustre: server umount lustre-MDT0000 complete [ 285.748355] LustreError: 15828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562423 with bad export cookie 14366380399670483026 [ 285.750944] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 285.754193] LustreError: 15828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 285.875871] Lustre: server umount lustre-MDT0001 complete [ 287.103674] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:00:23 (1783562423) [ 289.498727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 289.701639] LustreError: 19264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 289.734353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 291.266979] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 294.505931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 296.211678] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 299.163184] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 301.311582] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 304.231804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 306.920798] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 310.943236] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 319.972283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.975150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 319.985751] Lustre: Skipped 2 previous similar messages [ 319.987288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 319.989359] Lustre: Skipped 1 previous similar message [ 321.726965] Lustre: server umount lustre-OST0000 complete [ 325.485465] Lustre: server umount lustre-MDT0000 complete [ 326.894161] LustreError: 19245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562464 with bad export cookie 14366380399670483908 [ 326.897300] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 326.898035] LustreError: 19245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 327.035973] Lustre: server umount lustre-MDT0001 complete [ 331.133660] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 332.376150] Key type lgssc unregistered [ 332.529327] LNet: 22805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.532940] LNetError: 22805:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 332.543586] LNet: Removed LNI 192.168.203.136@tcp [ 332.944248] Key type .llcrypt unregistered [ 332.945882] Key type ._llcrypt unregistered [ 343.210862] Key type ._llcrypt registered [ 343.212566] Key type .llcrypt registered [ 343.257751] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 343.723922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 343.734318] alg: No test for adler32 (adler32-zlib) [ 344.615893] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 344.724097] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 346.319178] Key type lgssc registered [ 346.801415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 350.091289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 350.100741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 351.301815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 352.997166] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 357.103961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 357.317721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 359.229369] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 362.422756] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 364.731181] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 367.659492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 367.837404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 370.398293] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 373.609704] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 376.040899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 386.528362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 386.532286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.539433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 387.552681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 387.556439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 387.557292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 387.565247] Lustre: Skipped 1 previous similar message [ 388.441723] Lustre: server umount lustre-OST0000 complete [ 390.141669] Lustre: server umount lustre-MDT0000 complete [ 391.714768] LustreError: 24178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562529 with bad export cookie 10130674762864712328 [ 391.722160] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 391.723624] LustreError: 24178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 391.861753] Lustre: server umount lustre-MDT0001 complete [ 396.077081] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 397.373324] Key type lgssc unregistered [ 397.521455] LNet: 28166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.525950] LNetError: 28166:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 397.537305] LNet: Removed LNI 192.168.203.136@tcp [ 397.909806] Key type .llcrypt unregistered [ 397.912679] Key type ._llcrypt unregistered [ 407.978351] Key type ._llcrypt registered [ 407.979769] Key type .llcrypt registered [ 408.028329] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 408.563357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 408.573921] alg: No test for adler32 (adler32-zlib) [ 409.458464] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 409.568057] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 411.159236] Key type lgssc registered [ 411.622050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 414.897963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 414.907735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.120108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 417.876989] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 421.539295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.703754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 423.457804] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 426.500546] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 428.912964] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 434.755849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 434.927635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.319907] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 440.215206] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 441.070324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 456.673226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 456.678811] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.685757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.696832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.697829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.699379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.701636] Lustre: Skipped 1 previous similar message [ 459.690426] Lustre: server umount lustre-OST0000 complete [ 462.817488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.818583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 462.818855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 462.827563] Lustre: Skipped 1 previous similar message [ 467.567723] Lustre: server umount lustre-MDT0000 complete [ 467.936302] LustreError: 30793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.947854] LustreError: 30793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 469.217691] LustreError: 29535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562606 with bad export cookie 2433580849195318031 [ 469.220322] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 469.224140] LustreError: 29535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 469.384996] Lustre: server umount lustre-MDT0001 complete [ 473.790074] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 475.078901] Key type lgssc unregistered [ 475.236555] LNet: 33448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 475.240195] LNetError: 33448:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 475.250477] LNet: Removed LNI 192.168.203.136@tcp [ 475.617053] Key type .llcrypt unregistered [ 475.618844] Key type ._llcrypt unregistered [ 485.807781] Key type ._llcrypt registered [ 485.809400] Key type .llcrypt registered [ 485.868143] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 486.265626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 486.312944] alg: No test for adler32 (adler32-zlib) [ 487.199602] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 487.322228] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 488.935198] Key type lgssc registered [ 489.391940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 492.632954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 492.644495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.834935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.614808] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 499.392974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.568256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 501.283891] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 504.261407] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 506.463131] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 509.100298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 509.259380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 511.555606] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 514.647799] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 518.502722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 525.792520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 525.797514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.804895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.867174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.868156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.868936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 530.025989] Lustre: server umount lustre-OST0000 complete [ 533.985402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.989646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 533.997031] Lustre: Skipped 1 previous similar message [ 534.001091] Lustre: Skipped 1 previous similar message [ 537.838437] Lustre: server umount lustre-MDT0000 complete [ 539.104146] LustreError: 34854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.113919] LustreError: 34854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 539.446232] LustreError: 37413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562677 with bad export cookie 15078893593008475115 [ 539.449756] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 539.452563] LustreError: 37413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 539.583618] Lustre: server umount lustre-MDT0001 complete [ 543.991568] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 545.367109] Key type lgssc unregistered [ 545.519503] LNet: 38365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 545.525881] LNetError: 38365:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 545.538407] LNet: Removed LNI 192.168.203.136@tcp [ 545.923049] Key type .llcrypt unregistered [ 545.924729] Key type ._llcrypt unregistered [ 556.720145] Key type ._llcrypt registered [ 556.721834] Key type .llcrypt registered [ 556.770427] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 557.237677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 557.258908] alg: No test for adler32 (adler32-zlib) [ 558.145955] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 558.257837] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 559.855608] Key type lgssc registered [ 560.353708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 563.822104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 563.831733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.028345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 566.856868] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 570.643108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.820716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 572.498909] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 575.647033] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 578.096542] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 580.943428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 581.114172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 583.634096] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 586.668547] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 590.313985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 595.426652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.433758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 595.437074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 595.440274] Lustre: Skipped 1 previous similar message [ 597.478510] Lustre: lustre-OST0000: Not available for connect from 192.168.203.36@tcp (stopping) [ 600.546526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 600.549881] Lustre: Skipped 1 previous similar message [ 600.663615] Lustre: server umount lustre-OST0000 complete [ 614.188252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.196195] Lustre: Skipped 1 previous similar message [ 614.199689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 614.202435] Lustre: Skipped 1 previous similar message [ 619.488523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 619.491716] Lustre: Skipped 1 previous similar message [ 620.456292] Lustre: server umount lustre-MDT0000 complete [ 621.986153] LustreError: 42453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562759 with bad export cookie 2856787659442726675 [ 621.989331] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 621.992426] LustreError: 42453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 636.383460] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 636.479816] Lustre: server umount lustre-MDT0001 complete [ 640.947896] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 642.143763] Key type lgssc unregistered [ 642.292670] LNet: 43459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.296324] LNetError: 43459:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 642.306625] LNet: Removed LNI 192.168.203.136@tcp [ 642.680856] Key type .llcrypt unregistered [ 642.683136] Key type ._llcrypt unregistered [ 652.410710] Key type ._llcrypt registered [ 652.412341] Key type .llcrypt registered [ 652.453886] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 652.812225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 652.843045] alg: No test for adler32 (adler32-zlib) [ 653.722955] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 653.833992] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 655.423219] Key type lgssc registered [ 655.857053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 659.193169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 659.204617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 660.418095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.180616] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 665.969054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 666.169950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 667.963461] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 671.302235] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 673.738624] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 676.494462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 676.629598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 678.891256] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 681.881690] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 686.887082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 696.289790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 696.291629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 696.295460] Lustre: Skipped 2 previous similar messages [ 696.300604] Lustre: Skipped 2 previous similar messages [ 698.853273] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.36@tcp (stopping) [ 701.146238] Lustre: server umount lustre-MDT0000 complete [ 701.408856] LustreError: 44870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 701.416918] LustreError: 44870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 702.632290] LustreError: 44830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562840 with bad export cookie 10234297778313960679 [ 702.635298] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 702.638460] LustreError: 44830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 703.963702] LustreError: 44848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 703.978923] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.36@tcp (stopping) [ 706.528512] LustreError: 44848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.528597] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.543804] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 708.797794] Lustre: server umount lustre-MDT0001 complete [ 719.843609] LustreError: 46693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.36@tcp arrived at 1783562857 with bad export cookie 10234297778313961295 [ 719.849711] LustreError: 46693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 735.199162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 735.275816] Lustre: server umount lustre-OST0000 complete [ 741.791256] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 742.991270] Key type lgssc unregistered [ 743.135454] LNet: 48729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.139420] LNetError: 48729:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.155952] LNet: Removed LNI 192.168.203.136@tcp [ 743.502166] Key type .llcrypt unregistered [ 743.503508] Key type ._llcrypt unregistered [ 754.365394] Key type ._llcrypt registered [ 754.367014] Key type .llcrypt registered [ 754.406872] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 754.793469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 754.825121] alg: No test for adler32 (adler32-zlib) [ 755.710940] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 755.811577] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 757.399179] Key type lgssc registered [ 757.842626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 761.015800] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 761.025815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 776.479556] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 781.599411] LustreError: 50091:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.136@tcp: failed processing log, type 4: rc = -110 [ 808.223226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 810.553988] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 813.407640] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 976.734566] Lustre: server umount lustre-OST0000 complete [ 982.478803] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 983.517963] Key type lgssc unregistered [ 983.649452] LNet: 51766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.652724] LNetError: 51766:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.663460] LNet: Removed LNI 192.168.203.136@tcp [ 983.973802] Key type .llcrypt unregistered [ 983.975282] Key type ._llcrypt unregistered [ 993.083935] Key type ._llcrypt registered [ 993.085143] Key type .llcrypt registered [ 993.126248] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 993.545153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 993.554845] alg: No test for adler32 (adler32-zlib) [ 994.412755] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 994.507742] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 996.095128] Key type lgssc registered [ 996.465620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 999.062138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 999.068106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1000.203100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1001.538299] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1004.262669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.382855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1005.837726] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1008.187371] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1010.020732] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1012.230235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1012.342890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.220557] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1016.514808] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1019.872517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1019.875183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.879766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1023.369329] Lustre: server umount lustre-OST0000 complete [ 1024.992986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1024.993248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1024.994102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1025.005641] Lustre: Skipped 1 previous similar message [ 1030.112071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1030.114311] Lustre: Skipped 2 previous similar messages [ 1030.738797] Lustre: server umount lustre-MDT0000 complete [ 1032.010393] LustreError: 53138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563169 with bad export cookie 6243903074698494225 [ 1032.013326] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1032.015101] LustreError: 53138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1032.122270] Lustre: server umount lustre-MDT0001 complete [ 1035.710965] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 1036.750345] Key type lgssc unregistered [ 1036.887507] LNet: 56683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.890608] LNetError: 56683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.900338] LNet: Removed LNI 192.168.203.136@tcp [ 1037.231629] Key type .llcrypt unregistered [ 1037.233322] Key type ._llcrypt unregistered [ 1046.356635] Key type ._llcrypt registered [ 1046.358139] Key type .llcrypt registered [ 1046.400751] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1046.731601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1046.778747] alg: No test for adler32 (adler32-zlib) [ 1047.645751] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 1047.735431] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 1049.319165] Key type lgssc registered [ 1049.731290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1052.483840] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1052.493584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1067.935678] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1073.055414] LustreError: 58045:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.136@tcp: failed processing log, type 4: rc = -110 [ 1099.679162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1101.707826] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1104.226257] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1106.497061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1106.671405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1106.696373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1107.989917] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1110.897656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.018384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1112.406457] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1114.600543] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1116.342352] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1117.151928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1117.155288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.161684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1120.224616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1120.226757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.085041] Lustre: server umount lustre-OST0000 complete [ 1137.267287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.267926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.272515] Lustre: Skipped 1 previous similar message [ 1137.277808] Lustre: Skipped 2 previous similar messages [ 1140.384192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1140.386528] Lustre: Skipped 1 previous similar message [ 1143.449388] Lustre: server umount lustre-MDT0000 complete [ 1144.723276] LustreError: 59804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563282 with bad export cookie 10926745398884893300 [ 1144.726043] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1144.728476] LustreError: 59804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1159.135144] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1159.217802] Lustre: server umount lustre-MDT0001 complete [ 1162.861451] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 1163.926995] Key type lgssc unregistered [ 1164.050373] LNet: 61784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1164.053510] LNetError: 61784:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1164.061476] LNet: Removed LNI 192.168.203.136@tcp [ 1164.387975] Key type .llcrypt unregistered [ 1164.389638] Key type ._llcrypt unregistered [ 1172.803290] Key type ._llcrypt registered [ 1172.804709] Key type .llcrypt registered [ 1172.841375] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1173.152515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1173.216949] alg: No test for adler32 (adler32-zlib) [ 1174.063942] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 1174.153331] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 1175.735153] Key type lgssc registered [ 1176.093533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1178.751914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1178.758717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.899209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.227055] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1184.045842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.156873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1185.461239] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1187.935808] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1189.761061] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1192.236057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1192.376405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1194.419581] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1196.947350] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1205.671584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1210.849197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1210.850704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1210.852512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1210.856985] Lustre: Skipped 1 previous similar message [ 1211.465647] Lustre: server umount lustre-OST0000 complete [ 1214.804874] Lustre: server umount lustre-MDT0000 complete [ 1214.943764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1214.946401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.953146] LustreError: 63173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1214.960162] LustreError: 63173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1215.968529] LustreError: 63172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.968972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1216.020326] LustreError: 63153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563353 with bad export cookie 2137276079672271969 [ 1216.022383] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1216.025563] LustreError: 63153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1216.140575] Lustre: server umount lustre-MDT0001 complete [ 1219.504706] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 1220.499181] Key type lgssc unregistered [ 1220.614402] LNet: 66718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1220.617121] LNetError: 66718:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1220.625269] LNet: Removed LNI 192.168.203.136@tcp [ 1220.907538] Key type .llcrypt unregistered [ 1220.908805] Key type ._llcrypt unregistered [ 1230.415505] Key type ._llcrypt registered [ 1230.417135] Key type .llcrypt registered [ 1230.456600] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1230.837983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1230.907416] alg: No test for adler32 (adler32-zlib) [ 1231.756801] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 1231.843399] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 1233.423203] Key type lgssc registered [ 1233.800142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1236.612493] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 22:16:13 (1783563373) [ 1238.780079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1238.789506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.936042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.269956] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1243.972448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1244.097966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1245.404666] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1247.782408] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1249.692966] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1251.945488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1252.061425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1253.933834] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1256.243234] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1264.358024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1269.797592] Lustre: Failing over lustre-MDT0000 [ 1269.949875] Lustre: server umount lustre-MDT0000 complete [ 1272.802540] LustreError: 68392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1272.808894] LustreError: 68392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1272.841449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1272.876467] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1272.929090] Lustre: *** cfs_fail_loc=135, val=0*** [ 1272.971138] LustreError: 71499:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1272.973392] LustreError: 71499:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.136@tcp: cfg command failed: rc = -2 [ 1272.976354] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1272.976354] [ 1272.979600] LustreError: MGC192.168.203.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1272.983553] LustreError: 71487:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1272.986324] LustreError: 71487:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1272.989273] LustreError: 71487:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1272.991726] LustreError: 71487:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1273.003712] Lustre: server umount lustre-MDT0000 complete [ 1273.005089] LustreError: 71487:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1274.849587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1274.849734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1274.858179] LustreError: 68396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.863430] LustreError: 68396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1275.152838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.184838] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1275.270977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1275.295021] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1276.572472] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1277.915158] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1280.488269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1280.496034] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1280.511975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1285.088275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1285.091420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.096221] Lustre: Skipped 2 previous similar messages [ 1285.098582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1285.606536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1287.572535] Lustre: server umount lustre-OST0000 complete [ 1290.721911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.722205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1290.723308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.725683] Lustre: Skipped 2 previous similar messages [ 1290.729028] LustreError: Skipped 1 previous similar message [ 1294.927334] Lustre: server umount lustre-MDT0000 complete [ 1295.839879] LustreError: 68395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.845826] LustreError: 68395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1296.153299] LustreError: 72525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563433 with bad export cookie 2326595025739096718 [ 1296.154881] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1296.158541] LustreError: 72525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1296.273124] Lustre: server umount lustre-MDT0001 complete [ 1299.740455] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 1300.755736] Key type lgssc unregistered [ 1300.892341] LNet: 73277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1300.894512] LNetError: 73277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1300.901247] LNet: Removed LNI 192.168.203.136@tcp [ 1301.196988] Key type .llcrypt unregistered [ 1301.198846] Key type ._llcrypt unregistered [ 1310.058985] Key type ._llcrypt registered [ 1310.060599] Key type .llcrypt registered [ 1310.094668] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1310.395709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1310.440965] alg: No test for adler32 (adler32-zlib) [ 1311.285705] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 1311.366894] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 1312.943170] Key type lgssc registered [ 1313.288173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1315.888118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1315.894714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1317.031872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.313442] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1321.186321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1321.301763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1322.625349] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1325.049763] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1327.222401] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1329.609034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1329.723795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1331.751506] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1334.449610] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1342.053963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1347.145504] Lustre: Failing over lustre-MDT0000 [ 1347.311472] Lustre: server umount lustre-MDT0000 complete [ 1347.552650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.553625] LustreError: 77154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.558423] Lustre: Skipped 1 previous similar message [ 1347.565400] LustreError: 77154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1350.115193] LustreError: 77154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1350.287112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.326600] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1350.382334] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1350.431764] LustreError: 77773:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1350.435357] LustreError: 77773:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.136@tcp: cfg command failed: rc = -9 [ 1350.439831] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1350.439831] [ 1350.444789] LustreError: MGC192.168.203.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1350.450127] LustreError: 77761:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1350.453129] LustreError: 77761:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 1350.455448] LustreError: 77761:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1350.457243] LustreError: 77761:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1350.466901] Lustre: server umount lustre-MDT0000 complete [ 1350.469090] LustreError: 77761:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1352.159686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1352.162354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.166453] LustreError: 74666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.652415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.691627] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1352.768321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.789329] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1354.060866] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1355.226519] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1357.798236] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1357.804761] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1357.820941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1361.888466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1361.892600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.896435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1362.916896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1362.917503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1364.814262] Lustre: server umount lustre-OST0000 complete [ 1368.032679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1368.033053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1368.033671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1368.033675] Lustre: Skipped 1 previous similar message [ 1368.038210] Lustre: Skipped 2 previous similar messages [ 1373.152107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1373.155067] Lustre: Skipped 2 previous similar messages [ 1378.272350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1378.274089] Lustre: Skipped 1 previous similar message [ 1380.319126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1380.377174] Lustre: server umount lustre-MDT0000 complete [ 1381.558514] LustreError: 74649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563519 with bad export cookie 16016686055066696242 [ 1381.563142] LustreError: 74649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1381.563527] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1381.679936] Lustre: server umount lustre-MDT0001 complete [ 1385.475814] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 1386.546954] Key type lgssc unregistered [ 1386.670372] LNet: 79552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1386.673335] LNetError: 79552:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1386.680298] LNet: Removed LNI 192.168.203.136@tcp [ 1387.013979] Key type .llcrypt unregistered [ 1387.015231] Key type ._llcrypt unregistered [ 1390.829444] Key type ._llcrypt registered [ 1390.830467] Key type .llcrypt registered [ 1390.864025] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 1390.956224] Key type .llcrypt unregistered [ 1390.957276] Key type ._llcrypt unregistered [ 1399.378084] Key type ._llcrypt registered [ 1399.379685] Key type .llcrypt registered [ 1399.420515] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1399.815454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1399.831662] alg: No test for adler32 (adler32-zlib) [ 1400.688624] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 1400.773069] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 1402.351153] Key type lgssc registered [ 1402.724617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1405.205151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1405.210370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.339480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1407.625977] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1410.450513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.573777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1411.847665] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1413.984253] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1415.587571] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1417.638651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1417.757553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.728781] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1422.020646] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1432.102038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1436.130455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1436.132691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.136076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1437.153121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1437.154185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1437.155059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1441.170786] Lustre: server umount lustre-OST0000 complete [ 1446.367813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1446.370144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.373888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1446.377409] Lustre: Skipped 1 previous similar message [ 1448.531161] Lustre: server umount lustre-MDT0000 complete [ 1449.737353] LustreError: 81409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563587 with bad export cookie 9295620955679527519 [ 1449.741059] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1449.742969] LustreError: 81409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1449.858527] Lustre: server umount lustre-MDT0001 complete [ 1453.639125] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 1454.687846] Key type lgssc unregistered [ 1454.817414] LNet: 84980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.820129] LNetError: 84980:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.829335] LNet: Removed LNI 192.168.203.136@tcp [ 1455.162287] Key type .llcrypt unregistered [ 1455.163596] Key type ._llcrypt unregistered [ 1463.750643] Key type ._llcrypt registered [ 1463.751974] Key type .llcrypt registered [ 1463.799677] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1464.162240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1464.193063] alg: No test for adler32 (adler32-zlib) [ 1465.061278] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 1465.156442] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 1466.743148] Key type lgssc registered [ 1467.111551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1469.674521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1469.681302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.832829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.262861] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1475.197732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.322187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1476.742533] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1479.207041] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1480.932129] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1483.128768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1483.251145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.039070] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1487.275287] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1495.525703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1500.642122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.642232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1500.645702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1500.647250] LustreError: Skipped 1 previous similar message [ 1500.650039] Lustre: Skipped 1 previous similar message [ 1501.775661] Lustre: server umount lustre-OST0000 complete [ 1505.099353] Lustre: server umount lustre-MDT0000 complete [ 1505.760276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1505.765333] Lustre: Skipped 1 previous similar message [ 1505.767034] LustreError: 86388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.773163] LustreError: 86388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1506.214087] LustreError: 86351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563643 with bad export cookie 11359664167949270645 [ 1506.216238] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1506.217849] LustreError: 86351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1506.325107] Lustre: server umount lustre-MDT0001 complete [ 1509.607533] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 1510.526942] Key type lgssc unregistered [ 1510.648364] LNet: 89916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.650716] LNetError: 89916:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1510.659336] LNet: Removed LNI 192.168.203.136@tcp [ 1510.944976] Key type .llcrypt unregistered [ 1510.946216] Key type ._llcrypt unregistered [ 1519.206878] Key type ._llcrypt registered [ 1519.207707] Key type .llcrypt registered [ 1519.240935] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1519.686391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1519.699997] alg: No test for adler32 (adler32-zlib) [ 1520.539086] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 1520.612320] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 1522.191153] Key type lgssc registered [ 1522.503368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1524.851948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1524.859119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1526.002256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1527.275983] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1529.925950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1530.038303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1531.344067] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1533.630727] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1535.278308] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1537.351924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1537.455813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.251297] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1541.534617] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1551.780893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1554.912047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1554.915692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1554.920467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1556.962131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1556.962166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1556.962663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1560.905739] Lustre: server umount lustre-OST0000 complete [ 1562.080208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.080829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1562.085922] Lustre: Skipped 1 previous similar message [ 1562.088888] Lustre: Skipped 1 previous similar message [ 1567.199986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1567.201893] Lustre: Skipped 2 previous similar messages [ 1568.153705] Lustre: server umount lustre-MDT0000 complete [ 1569.300970] LustreError: 92038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563706 with bad export cookie 12866371518834480683 [ 1569.304193] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1569.304374] LustreError: 92038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1569.415680] Lustre: server umount lustre-MDT0001 complete [ 1572.709339] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 1573.664729] Key type lgssc unregistered [ 1573.792337] LNet: 94854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.794903] LNetError: 94854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.807291] LNet: Removed LNI 192.168.203.136@tcp [ 1574.090294] Key type .llcrypt unregistered [ 1574.091306] Key type ._llcrypt unregistered [ 1582.387376] Key type ._llcrypt registered [ 1582.388480] Key type .llcrypt registered [ 1582.428553] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1582.809845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1582.826914] alg: No test for adler32 (adler32-zlib) [ 1583.693255] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 1583.787376] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 1585.367160] Key type lgssc registered [ 1585.732669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1588.210508] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1588.216598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1603.615371] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1608.735259] LustreError: 96216:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.136@tcp: failed processing log, type 4: rc = -110 [ 1635.359297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.102321] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1639.338631] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1641.144546] Lustre: server umount lustre-OST0000 complete [ 1643.740284] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 22:23:00 (1783563780) [ 1645.772685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.923180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1647.193661] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1649.829624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1649.939941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1651.163691] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1653.332509] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1655.043219] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1657.166795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1657.279463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1659.071119] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1661.254519] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1670.632803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1675.749662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1675.750031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1675.759541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1675.762610] Lustre: Skipped 1 previous similar message [ 1677.077539] Lustre: server umount lustre-OST0000 complete [ 1680.407499] Lustre: server umount lustre-MDT0000 complete [ 1680.864919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1680.865512] LustreError: 98581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1680.869109] Lustre: Skipped 2 previous similar messages [ 1680.875699] LustreError: 98581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1680.880741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1681.611293] LustreError: 97836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563819 with bad export cookie 13666191172939310645 [ 1681.615524] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1681.616075] LustreError: 97836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1681.735122] Lustre: server umount lustre-MDT0001 complete [ 1684.948885] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 1685.876673] Key type lgssc unregistered [ 1685.992304] LNet: 101561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.995087] LNetError: 101561:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.005236] LNet: Removed LNI 192.168.203.136@tcp [ 1686.284104] Key type .llcrypt unregistered [ 1686.284994] Key type ._llcrypt unregistered [ 1700.398551] Key type ._llcrypt registered [ 1700.399556] Key type .llcrypt registered [ 1700.436144] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1700.757469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1700.798772] alg: No test for adler32 (adler32-zlib) [ 1701.652037] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 1701.744495] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 1703.327168] Key type lgssc registered [ 1703.688905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1706.295279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1706.302556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1707.466668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1708.827911] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1711.727605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1711.849764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1713.410864] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1716.109971] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1718.979610] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1726.749933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1727.148816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.975827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1736.007612] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1744.076906] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1753.577177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1753.621184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.640595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1757.159142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1757.159167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1757.175027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.179264] Lustre: Skipped 1 previous similar message [ 1759.620328] Lustre: server umount lustre-OST0000 complete [ 1763.299817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1763.317535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1763.329389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1767.393696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.395328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1767.408736] Lustre: Skipped 1 previous similar message [ 1769.232765] Lustre: server umount lustre-MDT0000 complete [ 1771.735684] LustreError: 103703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563909 with bad export cookie 17468906710035614381 [ 1771.743103] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1771.751303] LustreError: 103703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1772.022520] Lustre: server umount lustre-MDT0001 complete [ 1778.426523] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 1779.952076] Key type lgssc unregistered [ 1780.114291] LNet: 107251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.120333] LNetError: 107251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.129852] LNet: Removed LNI 192.168.203.136@tcp [ 1780.563250] Key type .llcrypt unregistered [ 1780.565374] Key type ._llcrypt unregistered [ 1791.481871] Key type ._llcrypt registered [ 1791.484080] Key type .llcrypt registered [ 1791.544642] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1792.081523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1792.119180] alg: No test for adler32 (adler32-zlib) [ 1793.072185] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 1793.230569] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 1794.847165] Key type lgssc registered [ 1795.486749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1799.940877] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1799.953428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1815.455452] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1820.575260] LustreError: 108472:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.136@tcp: failed processing log, type 4: rc = -110 [ 1847.199975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.001559] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1853.830457] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1857.316863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1857.465189] LustreError: MGC192.168.203.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1857.474868] LustreError: 109498:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1857.481065] LustreError: 109498:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1857.485175] LustreError: 109498:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1857.525236] Lustre: server umount lustre-MDT0000 complete [ 1857.527299] LustreError: 109498:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1870.871557] Lustre: server umount lustre-OST0000 complete [ 1876.018212] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 1881.417842] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1900.130404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.266864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1900.289322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1900.336983] Lustre: lustre-MDT0000: new disk, initializing [ 1900.388707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.399809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1902.562326] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1909.170666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.245984] Lustre: 112284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1909.292188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1909.296826] Lustre: Skipped 1 previous similar message [ 1909.342934] Lustre: lustre-MDT0001: new disk, initializing [ 1909.399233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1909.427613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1909.434719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1911.752739] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1917.204792] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1919.870966] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1923.543940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1923.681921] Lustre: lustre-OST0000: new disk, initializing [ 1923.685240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1923.690517] Lustre: 114250:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1923.733059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1927.029895] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1929.753110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1929.757854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1929.819577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1932.040554] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1934.276180] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1934.406231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1936.582819] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1936.689611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1939.941195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1939.944582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.948334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1939.955679] Lustre: Skipped 1 previous similar message [ 1939.969300] Lustre: Skipped 1 previous similar message [ 1943.811664] Lustre: server umount lustre-OST0000 complete [ 1950.177345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1950.184963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1950.191155] Lustre: Skipped 1 previous similar message [ 1950.202318] Lustre: Skipped 1 previous similar message [ 1952.365570] Lustre: server umount lustre-MDT0000 complete [ 1954.716123] LustreError: 112276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783564092 with bad export cookie 13335656205450706785 [ 1954.718711] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1954.727132] LustreError: 112276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1954.902641] Lustre: server umount lustre-MDT0001 complete [ 1960.546176] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 22:28:16 (1783564096) [ 1962.768487] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1973.562405] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 1979.192169] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 2000.426024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2000.559893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2000.591771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2000.650795] Lustre: lustre-MDT0000: new disk, initializing [ 2000.700578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2000.717553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2003.114965] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2009.306376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.371225] Lustre: 119145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2009.415844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2009.421164] Lustre: Skipped 1 previous similar message [ 2009.478322] Lustre: lustre-MDT0001: new disk, initializing [ 2009.541451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2009.565101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2009.570263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2011.954285] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2017.025060] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2020.335333] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2024.223137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2024.397430] Lustre: lustre-OST0000: new disk, initializing [ 2024.400770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2024.406126] Lustre: 121111:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2027.590258] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2032.657429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2032.662476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2032.683254] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2032.723857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2034.644606] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2034.747910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2036.678788] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2036.783881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2042.851741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2042.853687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.856413] LustreError: Skipped 2 previous similar messages [ 2042.864384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2042.867841] Lustre: Skipped 1 previous similar message [ 2047.970934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2052.063144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2052.146377] Lustre: server umount lustre-OST0000 complete [ 2055.648317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2055.651576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.655967] Lustre: Skipped 1 previous similar message [ 2055.658607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.662413] Lustre: Skipped 1 previous similar message [ 2060.199528] Lustre: server umount lustre-MDT0000 complete [ 2062.066729] LustreError: 119948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783564199 with bad export cookie 13335656205450707674 [ 2062.067631] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2062.073422] LustreError: 119948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2062.212948] Lustre: server umount lustre-MDT0001 complete [ 2067.104890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.441659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.446020] Lustre: Skipped 1 previous similar message [ 2069.756813] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2072.554777] LustreError: 123326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.563692] LustreError: 123326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2075.132669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.385991] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2082.856769] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2086.245191] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2090.590472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2094.470535] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2098.711669] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2103.449377] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2108.384833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2108.391051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.399380] Lustre: Skipped 1 previous similar message [ 2108.402782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2108.407012] Lustre: Skipped 2 previous similar messages [ 2111.653548] Lustre: server umount lustre-OST0000 complete [ 2119.750147] Lustre: server umount lustre-MDT0000 complete [ 2120.676444] LustreError: 125766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.693035] LustreError: 125766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2121.756689] LustreError: 123306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783564259 with bad export cookie 13335656205450708556 [ 2121.758518] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2121.764352] LustreError: 123306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2121.910629] Lustre: server umount lustre-MDT0001 complete [ 2128.011644] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 2129.595580] Key type lgssc unregistered [ 2129.770228] LNet: 126952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2129.779403] LNetError: 126952:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2129.789977] LNet: Removed LNI 192.168.203.136@tcp [ 2130.262301] Key type .llcrypt unregistered [ 2130.264602] Key type ._llcrypt unregistered [ 2140.612835] Key type ._llcrypt registered [ 2140.614947] Key type .llcrypt registered [ 2140.716192] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 2150.345480] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 2150.910278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2150.933954] alg: No test for adler32 (adler32-zlib) [ 2151.873083] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 2152.020620] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 2153.639157] Key type lgssc registered [ 2154.158024] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2173.391540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2173.401351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.541194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2174.573615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2174.630472] Lustre: lustre-MDT0000: new disk, initializing [ 2174.671824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2174.684861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2177.029871] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2183.426699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.516328] Lustre: 130078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2183.563752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2183.565802] Lustre: Skipped 1 previous similar message [ 2183.571446] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2183.616377] Lustre: lustre-MDT0001: new disk, initializing [ 2183.677532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2183.706588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2183.713994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2186.176695] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2191.992504] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2195.306464] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2199.373112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2199.515769] Lustre: lustre-OST0000: new disk, initializing [ 2199.519661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2199.525353] Lustre: 132045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2199.587398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2202.946503] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2208.784509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2208.791785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2208.809553] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2208.877368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2211.695307] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2211.805985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2214.364311] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2214.491502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2218.980553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2218.984239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2218.992829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2219.003794] Lustre: Skipped 1 previous similar message [ 2221.747951] Lustre: server umount lustre-OST0000 complete [ 2224.098077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2224.110543] Lustre: Skipped 1 previous similar message [ 2224.115361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2224.120485] Lustre: Skipped 1 previous similar message [ 2229.220072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2229.224137] Lustre: Skipped 1 previous similar message [ 2230.216760] Lustre: server umount lustre-MDT0000 complete [ 2232.551115] LustreError: 130073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783564370 with bad export cookie 4283192747056126474 [ 2232.557337] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2232.569932] LustreError: 130073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2232.774304] Lustre: server umount lustre-MDT0001 complete [ 2238.241657] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 22:32:54 (1783564374) [ 2242.087052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.384536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.720749] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2247.648808] LustreError: 134454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.660573] LustreError: 134454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2249.838263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.059670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2252.434275] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2256.298396] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2259.043899] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2260.016876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.023409] Lustre: Skipped 1 previous similar message [ 2260.026345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2260.030388] Lustre: Skipped 1 previous similar message [ 2271.712352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2271.715288] Lustre: Skipped 5 previous similar messages [ 2274.271117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2274.398446] Lustre: server umount lustre-MDT0000 complete [ 2275.817547] LustreError: 136022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.831109] LustreError: 136022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2276.583266] LustreError: 134435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783564414 with bad export cookie 4283192747056127363 [ 2276.591572] LustreError: 134435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2276.592509] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2276.770990] Lustre: server umount lustre-MDT0001 complete [ 2281.564572] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 22:33:37 (1783564417) [ 2285.578299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2301.087570] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2306.207488] LustreError: 136868:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.136@tcp: failed processing log, type 4: rc = -110 [ 2331.807345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2335.281470] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2339.554702] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2340.816707] Lustre: server umount lustre-OST0000 complete [ 2345.362390] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 22:34:41 (1783564481) [ 2348.893540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.149014] LustreError: 138312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.194032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.741094] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2354.655951] LustreError: 138313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.921129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.569623] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2363.572087] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2366.865828] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2370.912779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2374.424106] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2378.841356] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2391.521036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2391.529214] Lustre: Skipped 1 previous similar message [ 2391.530665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2391.537011] Lustre: Skipped 2 previous similar messages [ 2394.313346] Lustre: server umount lustre-MDT0000 complete [ 2396.483933] LustreError: 138292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783564534 with bad export cookie 4283192747056128014 [ 2396.488028] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2396.493256] LustreError: 138292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2396.641361] LustreError: 138309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.641763] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.704625] Lustre: server umount lustre-MDT0001 complete [ 2405.010981] Lustre: server umount lustre-OST0000 complete [ 2409.512812] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 22:35:45 (1783564545) [ 2413.222831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2413.463595] LustreError: 141961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.479462] LustreError: 141961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2413.520413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.524394] Lustre: Skipped 2 previous similar messages [ 2415.969261] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2420.835047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.569366] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2428.097748] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2431.457482] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2435.176106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2438.446607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2438.670308] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2443.034321] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2445.227529] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2445.372818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2447.576523] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2447.689744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2448.865887] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2448.872841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.884259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2454.769781] Lustre: server umount lustre-OST0000 complete [ 2457.057150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2471.391169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2471.515267] Lustre: server umount lustre-MDT0000 complete [ 2473.511817] LustreError: 143801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783564611 with bad export cookie 4283192747056129078 [ 2473.516243] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2473.523428] LustreError: 143801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2473.688341] Lustre: server umount lustre-MDT0001 complete [ 2478.146970] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 22:36:54 (1783564614) [ 2481.375323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2496.863643] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2501.983431] LustreError: 146116:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.136@tcp: failed processing log, type 4: rc = -110 [ 2527.583471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.588424] Lustre: Skipped 2 previous similar messages [ 2530.671131] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2534.376326] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2537.611655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2537.817565] LustreError: 146141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.829729] LustreError: 146141:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2537.893419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2540.018349] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2544.399627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.532885] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2549.798816] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2552.457836] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2554.555172] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2554.659545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2556.766302] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2556.881554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2559.973831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.975136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2559.978569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.978578] Lustre: Skipped 9 previous similar messages [ 2559.982300] Lustre: Skipped 5 previous similar messages [ 2572.255142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2572.344908] Lustre: server umount lustre-MDT0000 complete [ 2574.230288] LustreError: 146124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783564711 with bad export cookie 4283192747056130093 [ 2574.234239] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2574.238566] LustreError: 146124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2574.416628] Lustre: server umount lustre-MDT0001 complete [ 2585.592968] Lustre: server umount lustre-OST0000 complete [ 2589.637497] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 22:38:46 (1783564726) [ 2592.735402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2608.223495] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2613.343885] LustreError: 150307:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.136@tcp: failed processing log, type 4: rc = -110 [ 2641.988537] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2645.737885] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2649.040409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.257772] LustreError: 150332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.268460] LustreError: 150332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2649.356113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2651.339894] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2655.919594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.101297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2656.104328] Lustre: Skipped 4 previous similar messages [ 2658.172215] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2661.785811] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2664.690312] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2668.001895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2668.086942] Lustre: lustre-OST0001: new disk, initializing [ 2668.090353] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2668.094603] Lustre: 153188:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2668.229228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2668.236528] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2668.287466] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2671.275529] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2676.480784] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2678.688603] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2678.816597] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2681.155569] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2681.291668] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2683.876405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.883254] Lustre: Skipped 1 previous similar message [ 2688.399905] Lustre: server umount lustre-OST0000 complete [ 2694.115619] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2694.119278] LustreError: Skipped 1 previous similar message [ 2694.119712] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2694.128442] Lustre: Skipped 11 previous similar messages [ 2696.494436] Lustre: server umount lustre-OST0001 complete [ 2704.694714] Lustre: server umount lustre-MDT0000 complete [ 2706.515685] LustreError: 153187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783564844 with bad export cookie 4283192747056131045 [ 2706.523541] LustreError: 153187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2706.526048] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2706.683885] Lustre: server umount lustre-MDT0001 complete [ 2718.362823] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 22:40:54 (1783564854) [ 2719.207556] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2720.123202] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2720.919333] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 22:40:57 (1783564857) [ 2724.528767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2724.687532] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2724.703680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2726.941317] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2731.115648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2731.171888] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2731.194949] Lustre: 156743:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2733.478772] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2737.490342] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2740.478792] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2743.974900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2744.088894] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2747.100179] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2750.602452] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2752.555957] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2752.588752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2752.669506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2754.672773] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2754.773614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2757.602889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2757.603615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.617492] Lustre: Skipped 5 previous similar messages [ 2761.829773] Lustre: server umount lustre-OST0000 complete [ 2777.197086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2778.406060] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2778.412591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2778.412777] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2778.420359] Lustre: Skipped 1 previous similar message [ 2780.043228] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2783.383461] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2785.433458] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2785.527629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2787.194123] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2787.295991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2789.579505] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c0804e57000.ost_server_uuid 50 [ 2790.199288] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c0804e57000.ost_server_uuid in FULL state after 0 sec [ 2790.294095] LustreError: 156750:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.36@tcp) returned error from blocking AST (req@00000000728f3726 x1870202708112256 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000005ade4260/0x3b70f51c30cf7531 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.36@tcp remote: 0xaaa7c15750492768 expref: 5 pid: 156752 timeout: 2890 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2790.312149] LustreError: lustre-MDT0000: A client on nid 192.168.203.36@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2790.317149] LustreError: 156740:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.36@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000005ade4260/0x3b70f51c30cf7531 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.36@tcp remote: 0xaaa7c15750492768 expref: 6 pid: 156752 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2793.954651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2793.961204] LustreError: Skipped 2 previous similar messages [ 2797.879539] Lustre: server umount lustre-OST0000 complete [ 2805.756920] Lustre: server umount lustre-MDT0000 complete [ 2807.460311] LustreError: 156735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783564945 with bad export cookie 4283192747056132228 [ 2807.464083] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2807.465243] LustreError: 156735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2813.654465] Lustre: server umount lustre-MDT0001 complete [ 2818.342512] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 2819.567237] Key type lgssc unregistered [ 2819.707288] LNet: 162847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2819.713848] LNetError: 162847:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2819.722732] LNet: Removed LNI 192.168.203.136@tcp [ 2820.117202] Key type .llcrypt unregistered [ 2820.119124] Key type ._llcrypt unregistered [ 2831.494550] Key type ._llcrypt registered [ 2831.496015] Key type .llcrypt registered [ 2831.552196] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 2832.085523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2832.094317] alg: No test for adler32 (adler32-zlib) [ 2833.008500] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 2833.129677] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 2834.727156] Key type lgssc registered [ 2835.280695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2839.493755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2839.512640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2840.767888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2842.677103] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2846.948936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2847.145666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2849.105501] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2852.974808] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2855.793855] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2859.232060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2859.417954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.067580] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2865.580719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2865.842775] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2873.131582] Lustre: Failing over lustre-MDT0000 [ 2873.308343] Lustre: server umount lustre-MDT0000 complete [ 2876.898350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.901683] LustreError: 164234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2876.905366] Lustre: Skipped 2 previous similar messages [ 2876.916263] LustreError: 164234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2882.018694] LustreError: 164233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.028083] LustreError: 164233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2884.457241] LustreError: 164219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.36@tcp arrived at 1783565022 with bad export cookie 14045008914579823386 [ 2884.467159] LustreError: 164219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2887.137157] LustreError: 164237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.146769] LustreError: 164237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2887.910835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2887.981784] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2888.103519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.127525] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2890.100912] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2891.267408] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2891.274844] Lustre: lustre-MDT0000: Denying connection for new client b6fc5072-3700-45f5-91f8-20e381073b1b (at 192.168.203.36@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2893.287697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2898.401203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.403127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2898.410241] Lustre: Skipped 1 previous similar message [ 2898.417915] Lustre: Skipped 1 previous similar message [ 2903.522887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2903.527030] Lustre: Skipped 1 previous similar message [ 2904.423958] Lustre: server umount lustre-OST0000 complete [ 2906.173502] LustreError: 167959:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2906.177518] Lustre: 167309:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2906.183150] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2906.192161] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2906.199234] LustreError: 167309:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000a95d4f1 x1870203422020352/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2906.211287] LustreError: 167309:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2906.217220] LustreError: 167309:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2906.222444] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 2906.362881] Lustre: server umount lustre-MDT0000 complete [ 2908.080620] LustreError: 164218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565045 with bad export cookie 14045008914579823519 [ 2908.084088] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2908.088916] LustreError: 164218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2908.094865] LustreError: 163547:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2914.331445] Lustre: server umount lustre-MDT0001 complete [ 2919.434338] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 2920.760204] Key type lgssc unregistered [ 2920.902215] LNet: 168711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2920.906685] LNetError: 168711:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2920.915579] LNet: Removed LNI 192.168.203.136@tcp [ 2921.312866] Key type .llcrypt unregistered [ 2921.314703] Key type ._llcrypt unregistered [ 2932.616943] Key type ._llcrypt registered [ 2932.618674] Key type .llcrypt registered [ 2932.675676] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 2933.049660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2933.057804] alg: No test for adler32 (adler32-zlib) [ 2933.941950] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 2934.047407] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 2935.647179] Key type lgssc registered [ 2936.172707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2939.700136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2939.710895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.923347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2942.694620] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2946.280116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2946.463236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2948.244670] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2951.581605] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2954.010686] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2957.256251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2957.447257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2960.282182] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2963.862901] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2965.676771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2976.229081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2976.237140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2976.248184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2977.250130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2977.251060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2977.260889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2978.729158] Lustre: server umount lustre-OST0000 complete [ 2982.370363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2982.372802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2982.376804] Lustre: Skipped 1 previous similar message [ 2982.377088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2982.380534] Lustre: Skipped 2 previous similar messages [ 2986.861725] Lustre: server umount lustre-MDT0000 complete [ 2987.491333] LustreError: 170098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.499310] LustreError: 170098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2988.860446] LustreError: 172880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565126 with bad export cookie 7889548786457254953 [ 2988.864631] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2988.869518] LustreError: 172880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2989.078492] Lustre: server umount lustre-MDT0001 complete [ 2995.054421] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 2996.469654] Key type lgssc unregistered [ 2996.642944] LNet: 173631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2996.649951] LNetError: 173631:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2996.660281] LNet: Removed LNI 192.168.203.136@tcp [ 2997.099669] Key type .llcrypt unregistered [ 2997.101549] Key type ._llcrypt unregistered [ 3013.879018] Key type ._llcrypt registered [ 3013.880646] Key type .llcrypt registered [ 3013.929367] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3014.352790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3014.386795] alg: No test for adler32 (adler32-zlib) [ 3015.358760] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 3015.508369] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 3017.103280] Key type lgssc registered [ 3017.631438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3021.332567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3021.345815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3022.538784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3024.459224] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3028.805274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.007298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3030.905417] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3034.486557] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3037.353790] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3040.803399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3040.996471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.233706] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3048.172620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3048.347384] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3058.656809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3058.662147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3058.669110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3059.681882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3059.682606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3059.690705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3063.017346] Lustre: server umount lustre-OST0000 complete [ 3064.801172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3064.803909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.821422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3064.825210] Lustre: Skipped 2 previous similar messages [ 3069.921179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3069.925656] Lustre: Skipped 1 previous similar message [ 3075.040966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3075.045336] Lustre: Skipped 1 previous similar message [ 3079.135588] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3079.208156] Lustre: server umount lustre-MDT0000 complete [ 3080.161577] LustreError: 176038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3080.169798] LustreError: 176038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3081.018198] LustreError: 175291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565218 with bad export cookie 12979777205365409753 [ 3081.020213] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3081.025509] LustreError: 175291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3081.203107] Lustre: server umount lustre-MDT0001 complete [ 3085.716280] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 3087.159431] Key type lgssc unregistered [ 3087.315770] LNet: 178841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3087.321749] LNetError: 178841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3088.357170] LNet: Removed LNI 192.168.203.136@tcp [ 3088.838636] Key type .llcrypt unregistered [ 3088.840439] Key type ._llcrypt unregistered [ 3099.597107] Key type ._llcrypt registered [ 3099.598677] Key type .llcrypt registered [ 3099.663137] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3100.171069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3100.185289] alg: No test for adler32 (adler32-zlib) [ 3101.100809] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 3101.239031] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 3102.839185] Key type lgssc registered [ 3103.363570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3108.112176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3108.123763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3109.309905] Lustre: *** cfs_fail_loc=135, val=0*** [ 3109.357267] LustreError: 180340:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3109.360282] LustreError: 180340:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.136@tcp: cfg command failed: rc = -2 [ 3109.363958] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3109.363958] [ 3109.368328] LustreError: MGC192.168.203.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3109.375178] LustreError: 180302:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3109.379476] LustreError: 180302:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3109.385970] LustreError: 180302:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3109.390213] LustreError: 180302:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3109.427990] Lustre: server umount lustre-MDT0000 complete [ 3109.430159] LustreError: 180302:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3112.408411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.616641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3114.324435] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3117.775307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.943136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3119.660291] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3122.783817] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3125.135790] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3128.035838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.043746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3128.049171] Lustre: Skipped 1 previous similar message [ 3132.145996] Lustre: server umount lustre-MDT0000 complete [ 3133.408494] LustreError: 180711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.415179] LustreError: 180711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3133.612189] LustreError: 180691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565271 with bad export cookie 9242194610544853378 [ 3133.614617] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3133.617458] LustreError: 180691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3133.731949] Lustre: server umount lustre-MDT0001 complete [ 3137.715095] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 3138.874855] Key type lgssc unregistered [ 3139.024513] LNet: 183029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3139.028158] LNetError: 183029:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3139.038409] LNet: Removed LNI 192.168.203.136@tcp [ 3139.404781] Key type .llcrypt unregistered [ 3139.406414] Key type ._llcrypt unregistered [ 3148.427988] Key type ._llcrypt registered [ 3148.429742] Key type .llcrypt registered [ 3148.469278] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 3148.613478] Key type .llcrypt unregistered [ 3148.615556] Key type ._llcrypt unregistered [ 3156.319722] Key type ._llcrypt registered [ 3156.321119] Key type .llcrypt registered [ 3156.369306] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3156.738677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3156.760087] alg: No test for adler32 (adler32-zlib) [ 3157.642408] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 3157.745685] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 3159.335233] Key type lgssc registered [ 3159.859354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3163.140726] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3163.148347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3178.591533] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3183.711279] LustreError: 185136:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.136@tcp: failed processing log, type 4: rc = -110 [ 3210.336074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.579055] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3215.482791] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3218.032043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3218.224560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3218.252720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3219.819769] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3222.912861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3223.054973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3224.619431] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3227.786086] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3229.992122] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3231.112304] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3238.368653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3238.372232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3238.378816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3238.879885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3238.884592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3238.893242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3240.418418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3240.423169] Lustre: Skipped 1 previous similar message [ 3243.674587] Lustre: server umount lustre-OST0000 complete [ 3245.538746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.540033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3245.544049] Lustre: Skipped 1 previous similar message [ 3245.549082] Lustre: Skipped 1 previous similar message [ 3250.656345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3250.661021] Lustre: Skipped 1 previous similar message [ 3251.361954] Lustre: server umount lustre-MDT0000 complete [ 3252.937395] LustreError: 187793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565390 with bad export cookie 6116322152639611933 [ 3252.940957] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3252.943159] LustreError: 187793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3253.081758] Lustre: server umount lustre-MDT0001 complete [ 3257.262539] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 3258.416881] Key type lgssc unregistered [ 3258.549713] LNet: 189096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3258.553383] LNetError: 189096:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3258.560495] LNet: Removed LNI 192.168.203.136@tcp [ 3258.887085] Key type .llcrypt unregistered [ 3258.888893] Key type ._llcrypt unregistered [ 3269.314226] Key type ._llcrypt registered [ 3269.315736] Key type .llcrypt registered [ 3269.362314] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3269.711745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3269.759132] alg: No test for adler32 (adler32-zlib) [ 3270.628465] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 3270.727461] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 3272.319174] Key type lgssc registered [ 3272.756880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3275.796619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3275.804461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3276.976586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3278.431256] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3281.500255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3281.648741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3283.093380] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3285.702666] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3287.751446] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3290.276698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3290.404867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3292.350066] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3294.823966] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3301.673235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3306.651875] Lustre: Failing over lustre-MDT0000 [ 3306.743870] Lustre: server umount lustre-MDT0000 complete [ 3306.977389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3306.978650] LustreError: 190481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3306.982494] Lustre: Skipped 2 previous similar messages [ 3306.989959] LustreError: 190481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3310.562338] LustreError: 190481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.097089] LustreError: 190484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.103369] LustreError: 190484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3315.674692] LustreError: 192486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.663040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.701378] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3319.802478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3319.823651] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3320.794728] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3321.162274] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3322.678608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3324.902336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3324.907897] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3324.926160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3328.503408] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3340.259213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.264627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3340.264873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3340.269400] Lustre: Skipped 1 previous similar message [ 3343.888457] Lustre: server umount lustre-OST0000 complete [ 3345.376629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.376724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3345.377239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3345.380447] Lustre: Skipped 2 previous similar messages [ 3350.495959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3350.498398] Lustre: Skipped 2 previous similar messages [ 3351.184716] Lustre: server umount lustre-MDT0000 complete [ 3352.375377] LustreError: 190465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565490 with bad export cookie 12321095807877316401 [ 3352.378739] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3352.379391] LustreError: 190465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3352.489887] Lustre: server umount lustre-MDT0001 complete [ 3355.781410] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 3356.683374] Key type lgssc unregistered [ 3356.807309] LNet: 195431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3356.809665] LNetError: 195431:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3356.817366] LNet: Removed LNI 192.168.203.136@tcp [ 3357.096918] Key type .llcrypt unregistered [ 3357.097934] Key type ._llcrypt unregistered [ 3365.498912] Key type ._llcrypt registered [ 3365.500662] Key type .llcrypt registered [ 3365.541232] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3365.842688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3365.871027] alg: No test for adler32 (adler32-zlib) [ 3366.716238] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 3366.795515] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 3368.375115] Key type lgssc registered [ 3368.679853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3371.002466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3371.008375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.134198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3373.321168] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3375.926911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.038140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3377.240726] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3379.496592] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3381.121263] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3383.226778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3383.336117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3385.106841] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3387.392945] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3396.648572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3398.691223] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3401.106297] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3409.556069] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3416.033736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3416.036640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.041036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.058651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.058714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.058797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3417.068614] Lustre: Skipped 1 previous similar message [ 3420.686906] Lustre: server umount lustre-OST0000 complete [ 3422.176854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3422.177121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.181081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3422.185289] Lustre: Skipped 1 previous similar message [ 3427.296555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.299017] Lustre: Skipped 2 previous similar messages [ 3428.117750] Lustre: server umount lustre-MDT0000 complete [ 3429.347141] LustreError: 196802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565567 with bad export cookie 11768265666153780737 [ 3429.351110] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3429.352611] LustreError: 196802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3429.467608] Lustre: server umount lustre-MDT0001 complete [ 3433.152801] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 3434.170144] Key type lgssc unregistered [ 3434.293436] LNet: 200685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3434.297178] LNetError: 200685:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3434.306324] LNet: Removed LNI 192.168.203.136@tcp [ 3434.638159] Key type .llcrypt unregistered [ 3434.639822] Key type ._llcrypt unregistered [ 3442.417500] Key type ._llcrypt registered [ 3442.418462] Key type .llcrypt registered [ 3442.454229] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3442.825744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3442.864467] alg: No test for adler32 (adler32-zlib) [ 3443.733638] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 3443.826303] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 3445.407126] Key type lgssc registered [ 3445.757590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3448.239120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3448.245804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.395335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3450.749116] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3453.588806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.710600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3455.040202] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3457.578307] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3459.512487] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3461.836067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3461.964280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3464.062083] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3466.604692] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3475.302222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3478.414633] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3487.367164] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3494.160832] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3498.083248] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3510.240319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3510.244520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3510.250142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3511.266155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3511.267138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3511.268047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3511.276404] Lustre: Skipped 1 previous similar message [ 3513.359966] Lustre: server umount lustre-OST0000 complete [ 3515.359970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3515.363252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.369078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3520.725416] Lustre: server umount lustre-MDT0000 complete [ 3521.505348] LustreError: 201945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3521.511054] LustreError: 201945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3522.009066] LustreError: 201909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565659 with bad export cookie 769402089807916725 [ 3522.012589] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3522.014310] LustreError: 201909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3522.126114] Lustre: server umount lustre-MDT0001 complete [ 3525.601201] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 3526.595818] Key type lgssc unregistered [ 3526.729346] LNet: 207249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3526.732137] LNetError: 207249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3526.741342] LNet: Removed LNI 192.168.203.136@tcp [ 3527.042898] Key type .llcrypt unregistered [ 3527.044250] Key type ._llcrypt unregistered [ 3536.394261] Key type ._llcrypt registered [ 3536.395593] Key type .llcrypt registered [ 3536.436263] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3536.755402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3536.785067] alg: No test for adler32 (adler32-zlib) [ 3537.640669] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 3537.731638] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 3539.319125] Key type lgssc registered [ 3539.708857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3542.399902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3542.406060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3543.539889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3544.901350] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3547.838037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3547.963250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3549.309284] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3551.886103] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3553.831644] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3556.158190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3556.287539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3558.306105] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3560.964801] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3568.615225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3577.823790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3577.826127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3577.830709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3578.849640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3578.850818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3578.851107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3581.077294] Lustre: server umount lustre-OST0000 complete [ 3583.968716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3583.968737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3583.969690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3583.969695] Lustre: Skipped 1 previous similar message [ 3583.980060] Lustre: Skipped 1 previous similar message [ 3588.562866] Lustre: server umount lustre-MDT0000 complete [ 3589.087941] LustreError: 210067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3589.092602] LustreError: 210067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3589.785439] LustreError: 208619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565727 with bad export cookie 5188993424909301358 [ 3589.788652] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3589.792795] LustreError: 208619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3589.913731] Lustre: server umount lustre-MDT0001 complete [ 3593.553555] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 3594.592881] Key type lgssc unregistered [ 3594.724425] LNet: 212334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3594.727764] LNetError: 212334:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3594.738398] LNet: Removed LNI 192.168.203.136@tcp [ 3595.037768] Key type .llcrypt unregistered [ 3595.038841] Key type ._llcrypt unregistered [ 3602.340453] Key type ._llcrypt registered [ 3602.341952] Key type .llcrypt registered [ 3602.381931] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3602.744068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3602.757696] alg: No test for adler32 (adler32-zlib) [ 3603.612972] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 3603.699799] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 3605.287158] Key type lgssc registered [ 3605.667634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3608.425891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3608.434022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3609.596469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3611.042802] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3614.129976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.276166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3615.771241] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3618.356637] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3620.239031] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3622.582890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3622.725409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3624.847236] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3627.481762] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3635.052185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3644.384060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3644.387381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3644.392307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3645.408738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3645.410961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3645.413846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3647.834190] Lustre: server umount lustre-OST0000 complete [ 3650.015933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3650.018288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.022647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3650.025246] Lustre: Skipped 1 previous similar message [ 3655.446313] Lustre: server umount lustre-MDT0000 complete [ 3655.648226] LustreError: 213578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3655.653663] LustreError: 213578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3656.661039] LustreError: 213603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565794 with bad export cookie 3453582247750514274 [ 3656.664264] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3656.665121] LustreError: 213603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3656.774151] Lustre: server umount lustre-MDT0001 complete [ 3660.368808] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 3661.335762] Key type lgssc unregistered [ 3661.449367] LNet: 217276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3661.451651] LNetError: 217276:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3661.458255] LNet: Removed LNI 192.168.203.136@tcp [ 3661.742683] Key type .llcrypt unregistered [ 3661.744217] Key type ._llcrypt unregistered [ 3668.816733] Key type ._llcrypt registered [ 3668.818453] Key type .llcrypt registered [ 3668.868447] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3669.246862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3669.285070] alg: No test for adler32 (adler32-zlib) [ 3670.139742] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 3670.228628] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 3671.815215] Key type lgssc registered [ 3672.250738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3675.022119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3675.028411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3676.172577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3677.623618] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3680.818297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3680.952677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3682.373213] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3684.788098] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3686.753409] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3689.087292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3689.210132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.416496] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3694.156100] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3700.457837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3707.554688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3707.584979] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3707.658587] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3709.703283] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3712.305546] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3712.883348] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3723.481101] Lustre: Permanently deactivating lustre-OST0001 [ 3723.487499] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3730.423073] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3733.336565] Lustre: Permanently reactivating lustre-OST0001 [ 3733.344563] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3733.346796] Lustre: Skipped 2 previous similar messages [ 3738.613757] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3738.617968] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3738.620364] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3738.623666] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3738.635405] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3743.712196] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3743.714790] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.718890] Lustre: Skipped 1 previous similar message [ 3743.721203] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3743.723307] Lustre: Skipped 1 previous similar message [ 3745.249823] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3745.251785] Lustre: Skipped 1 previous similar message [ 3746.556197] Lustre: server umount lustre-OST0001 complete [ 3750.369728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.370845] LustreError: 220658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3750.373636] Lustre: Skipped 2 previous similar messages [ 3750.374120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3750.379428] LustreError: 220658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3750.380747] Lustre: Skipped 1 previous similar message [ 3755.489483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3755.490446] LustreError: 220338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3755.491319] Lustre: Skipped 1 previous similar message [ 3755.495718] LustreError: 220338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3760.611745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3760.612045] LustreError: 220338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3760.613489] Lustre: Skipped 1 previous similar message [ 3760.618296] LustreError: 220338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3762.143137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3762.207566] Lustre: server umount lustre-OST0000 complete [ 3765.728610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3765.733316] Lustre: Skipped 1 previous similar message [ 3769.625328] Lustre: server umount lustre-MDT0000 complete [ 3770.848909] LustreError: 218518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3770.854083] LustreError: 218518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3770.931989] LustreError: 218499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565908 with bad export cookie 7654144731107381372 [ 3770.934798] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3770.937164] LustreError: 218499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3771.064279] Lustre: server umount lustre-MDT0001 complete [ 3775.095403] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 3776.186887] Key type lgssc unregistered [ 3776.324529] LNet: 223573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3776.327688] LNetError: 223573:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3776.337419] LNet: Removed LNI 192.168.203.136@tcp [ 3776.658948] Key type .llcrypt unregistered [ 3776.660823] Key type ._llcrypt unregistered [ 3791.991626] Key type ._llcrypt registered [ 3791.993086] Key type .llcrypt registered [ 3792.034735] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3792.563260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3792.592862] alg: No test for adler32 (adler32-zlib) [ 3793.464202] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 3793.566420] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 3795.167154] Key type lgssc registered [ 3795.555310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3798.275617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3798.285512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3799.378464] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3799.385592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3799.444400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3800.840506] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3803.740198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3803.777081] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3803.789509] Lustre: 225753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3803.889839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3805.222525] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3807.680341] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3809.559161] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3811.915188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3811.984437] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3812.064612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.109877] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3816.694448] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3819.539296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3824.502417] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3830.429617] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3830.433729] Lustre: Skipped 1 previous similar message [ 3838.009623] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3846.610512] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3855.278095] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3874.657335] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3874.659896] Lustre: Skipped 1 previous similar message [ 3908.119971] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3908.123186] Lustre: Skipped 3 previous similar messages [ 3973.378785] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3973.381153] Lustre: Skipped 6 previous similar messages [ 4003.811449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4003.811917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4003.814944] LustreError: Skipped 1 previous similar message [ 4003.816224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4003.820196] Lustre: Skipped 1 previous similar message [ 4006.225520] Lustre: server umount lustre-OST0000 complete [ 4008.928642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.928743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4008.929633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4008.929638] Lustre: Skipped 1 previous similar message [ 4008.932948] Lustre: Skipped 1 previous similar message [ 4013.664054] Lustre: server umount lustre-MDT0000 complete [ 4014.939284] LustreError: 225742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566152 with bad export cookie 13324915094728123880 [ 4014.943071] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4014.944695] LustreError: 225742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4015.055589] Lustre: server umount lustre-MDT0001 complete [ 4018.915333] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 4019.936937] Key type lgssc unregistered [ 4020.085351] LNet: 230599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4020.089231] LNetError: 230599:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4020.099318] LNet: Removed LNI 192.168.203.136@tcp [ 4020.398320] Key type .llcrypt unregistered [ 4020.399968] Key type ._llcrypt unregistered [ 4029.655324] Key type ._llcrypt registered [ 4029.656990] Key type .llcrypt registered [ 4029.697264] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 4030.040416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4030.054844] alg: No test for adler32 (adler32-zlib) [ 4030.929711] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 4031.073537] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 4032.671455] Key type lgssc registered [ 4033.122808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4035.935640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4035.944780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4037.089915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.496399] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4041.272904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4041.394066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4042.714236] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4045.033829] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4046.897069] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4049.242836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4049.361628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4051.106653] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4053.331984] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4061.674872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 4071.909905] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4071.910438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4071.911325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4071.913232] LustreError: Skipped 1 previous similar message [ 4071.919102] Lustre: Skipped 1 previous similar message [ 4075.919825] Lustre: server umount lustre-OST0000 complete [ 4077.535735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4077.538029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4077.543957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4077.545585] Lustre: Skipped 1 previous similar message [ 4082.144501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4082.145394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.284414] Lustre: server umount lustre-MDT0000 complete [ 4084.525727] LustreError: 231971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566222 with bad export cookie 9395492240049943600 [ 4084.530560] LustreError: 231971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4084.530860] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4084.650734] Lustre: server umount lustre-MDT0001 complete [ 4088.427379] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 4089.480396] Key type lgssc unregistered [ 4089.640474] LNet: 235692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4089.642913] LNetError: 235692:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4089.651360] LNet: Removed LNI 192.168.203.136@tcp [ 4089.993922] Key type .llcrypt unregistered [ 4089.995693] Key type ._llcrypt unregistered [ 4098.806038] Key type ._llcrypt registered [ 4098.807073] Key type .llcrypt registered [ 4098.844270] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 4099.175813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4099.222981] alg: No test for adler32 (adler32-zlib) [ 4100.085327] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 4100.176831] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 4101.759175] Key type lgssc registered [ 4102.189763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4366.615433] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 4367.597076] Key type lgssc unregistered [ 4367.722290] LNet: 237691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4367.725108] LNetError: 237691:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4367.734259] LNet: Removed LNI 192.168.203.136@tcp [ 4368.031462] Key type .llcrypt unregistered [ 4368.032693] Key type ._llcrypt unregistered [ 4376.199977] Key type ._llcrypt registered [ 4376.202098] Key type .llcrypt registered [ 4376.240461] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 4382.919034] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 4383.232784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4383.265772] alg: No test for adler32 (adler32-zlib) [ 4384.125736] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 4384.218137] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 4385.807169] Key type lgssc registered [ 4386.172805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4396.684091] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4220 sec ========== 23:08:53 (1783566533) [ 4397.284236] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:08:53 (1783566533) === [ 4398.399465] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:08:54 (1783566534) === [ 4411.324116] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 4412.390251] Key type lgssc unregistered [ 4412.526279] LNet: 242895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4412.528529] LNetError: 242895:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4412.536274] LNet: Removed LNI 192.168.203.136@tcp [ 4412.848292] Key type .llcrypt unregistered [ 4412.849469] Key type ._llcrypt unregistered