[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 419774656 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002262] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.009187] LSM: Security Framework initializing [ 0.010087] Yama: becoming mindful. [ 0.012051] SELinux: Initializing. [ 0.013085] *** VALIDATE selinux *** [ 0.022107] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026381] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027186] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028147] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030036] *** VALIDATE tmpfs *** [ 0.031476] *** VALIDATE proc *** [ 0.033191] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.035230] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036133] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037023] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.041896] debug: unmapping init [mem 0xffffffff9c259000-0xffffffff9c260fff] [ 0.043749] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044787] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045046] ... version: 2 [ 0.045960] ... bit width: 48 [ 0.046008] ... generic registers: 4 [ 0.046851] ... value mask: 0000ffffffffffff [ 0.047013] ... max period: 00007fffffffffff [ 0.048017] ... fixed-purpose events: 3 [ 0.049010] ... event mask: 000000070000000f [ 0.050345] rcu: Hierarchical SRCU implementation. [ 0.052624] smp: Bringing up secondary CPUs ... [ 0.053415] x86: Booting SMP configuration: [ 0.054011] .... node #0, CPUs: #1 #2 #3 [ 0.058702] smp: Brought up 1 node, 4 CPUs [ 0.060019] smpboot: Max logical packages: 1 [ 0.061011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.200324] node 0 deferred pages initialised in 139ms [ 0.209191] devtmpfs: initialized [ 0.210634] x86/mm: Memory block size: 128MB [ 0.214121] gcov: version magic: 0x41383552 [ 0.217334] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.220077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.223317] pinctrl core: initialized pinctrl subsystem [ 0.225166] [ 0.225750] ************************************************************* [ 0.227011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.230014] ** ** [ 0.235036] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.240020] ** ** [ 0.244154] ** This means that this kernel is built to expose internal ** [ 0.246017] ** IOMMU data structures, which may compromise security on ** [ 0.250021] ** your system. ** [ 0.251010] ** ** [ 0.253012] ** If you see this message and you are not debugging the ** [ 0.255014] ** kernel, report this immediately to your vendor! ** [ 0.258010] ** ** [ 0.260016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262015] ************************************************************* [ 0.264475] NET: Registered protocol family 16 [ 0.267269] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.270069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.273066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.277009] cpuidle: using governor menu [ 0.283520] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.286745] PCI: Using configuration type 1 for base access [ 0.289288] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.296058] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.298062] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.304273] cryptd: max_cpu_qlen set to 1000 [ 0.306456] ACPI: Added _OSI(Module Device) [ 0.308031] ACPI: Added _OSI(Processor Device) [ 0.309009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.310007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.314363] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.322852] ACPI: Interpreter enabled [ 0.324061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.326017] ACPI: Using IOAPIC for interrupt routing [ 0.329136] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.333524] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.345778] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.347028] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.348011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.351085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.355388] acpiphp: Slot [2] registered [ 0.358193] acpiphp: Slot [5] registered [ 0.359098] acpiphp: Slot [6] registered [ 0.361100] acpiphp: Slot [7] registered [ 0.362121] acpiphp: Slot [8] registered [ 0.364103] acpiphp: Slot [9] registered [ 0.365133] acpiphp: Slot [10] registered [ 0.367099] acpiphp: Slot [3] registered [ 0.368080] acpiphp: Slot [4] registered [ 0.370138] acpiphp: Slot [11] registered [ 0.372095] acpiphp: Slot [12] registered [ 0.374204] acpiphp: Slot [13] registered [ 0.376084] acpiphp: Slot [14] registered [ 0.378126] acpiphp: Slot [15] registered [ 0.381099] acpiphp: Slot [16] registered [ 0.383113] acpiphp: Slot [17] registered [ 0.384103] acpiphp: Slot [18] registered [ 0.386148] acpiphp: Slot [19] registered [ 0.387094] acpiphp: Slot [20] registered [ 0.388088] acpiphp: Slot [21] registered [ 0.389150] acpiphp: Slot [22] registered [ 0.390088] acpiphp: Slot [23] registered [ 0.391218] acpiphp: Slot [24] registered [ 0.393085] acpiphp: Slot [25] registered [ 0.394099] acpiphp: Slot [26] registered [ 0.396103] acpiphp: Slot [27] registered [ 0.398156] acpiphp: Slot [28] registered [ 0.400120] acpiphp: Slot [29] registered [ 0.401168] acpiphp: Slot [30] registered [ 0.404203] acpiphp: Slot [31] registered [ 0.408104] PCI host bridge to bus 0000:00 [ 0.409036] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.413027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.417029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.421030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.423020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.426026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.427194] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.430145] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.433310] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.443016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.451020] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.454023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.457021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.460021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.464734] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.472035] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.475098] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.478940] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.485018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.499018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.504016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.509519] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.516018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.524015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.540022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.548293] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.556019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.562017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.580020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.590381] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.597069] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.601016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.619020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.629264] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.634014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.638017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.653016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.665578] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.674020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.678019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.700014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.715982] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.727018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.732000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.749021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.769149] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.772492] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.779918] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.784535] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.787337] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.794064] iommu: Default domain type: Passthrough [ 0.795569] SCSI subsystem initialized [ 0.797174] ACPI: bus type USB registered [ 0.798065] usbcore: registered new interface driver usbfs [ 0.799048] usbcore: registered new interface driver hub [ 0.803127] usbcore: registered new device driver usb [ 0.806383] pps_core: LinuxPPS API ver. 1 registered [ 0.808012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.811063] PTP clock support registered [ 0.813248] EDAC MC: Ver: 3.0.0 [ 0.816259] PCI: Using ACPI for IRQ routing [ 0.819264] NetLabel: Initializing [ 0.822116] NetLabel: domain hash size = 128 [ 0.824012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.827125] NetLabel: unlabeled traffic allowed by default [ 0.831067] vgaarb: loaded [ 0.836186] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.840017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.849508] clocksource: Switched to clocksource kvm-clock [ 0.973991] VFS: Disk quotas dquot_6.6.0 [ 0.975140] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.977117] *** VALIDATE ramfs *** [ 0.978752] *** VALIDATE hugetlbfs *** [ 0.980895] pnp: PnP ACPI init [ 0.983774] pnp: PnP ACPI: found 6 devices [ 1.003530] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.007382] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.009932] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.012589] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.016416] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.018727] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.022146] NET: Registered protocol family 2 [ 1.024963] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.030568] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.033680] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.038488] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.043579] TCP: Hash tables configured (established 65536 bind 65536) [ 1.048030] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.050920] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.053651] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.056675] NET: Registered protocol family 1 [ 1.060633] RPC: Registered named UNIX socket transport module. [ 1.063470] RPC: Registered udp transport module. [ 1.067959] RPC: Registered tcp transport module. [ 1.072833] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.077536] NET: Registered protocol family 44 [ 1.079116] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.080967] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.082776] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.084908] PCI: CLS 0 bytes, default 64 [ 1.086968] Unpacking initramfs... [ 2.774103] debug: unmapping init [mem 0xffff8a4b3cc54000-0xffff8a4b3ffbffff] [ 2.782635] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.785364] software IO TLB: mapped [mem 0x00000000a0400000-0x00000000a4400000] (64MB) [ 2.790612] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.427197] Initialise system trusted keyrings [ 3.438721] Key type blacklist registered [ 3.465780] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.490024] zbud: loaded [ 3.498578] *** VALIDATE nfs *** [ 3.500030] *** VALIDATE nfs4 *** [ 3.501663] pstore: using deflate compression [ 3.512801] Platform Keyring initialized [ 3.675792] NET: Registered protocol family 38 [ 3.678024] Key type asymmetric registered [ 3.679446] Asymmetric key parser 'x509' registered [ 3.682336] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.685861] io scheduler mq-deadline registered [ 3.687985] io scheduler kyber registered [ 3.690216] io scheduler bfq registered [ 3.692417] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.696034] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.699592] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.703234] ACPI: Power Button [PWRF] [ 3.708780] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.716394] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.731933] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.745365] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.775645] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.809459] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.848491] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.855516] Non-volatile memory driver v1.3 [ 3.857475] Linux agpgart interface v0.103 [ 3.884678] virtio_blk virtio1: [vda] 139280 512-byte logical blocks (71.3 MB/68.0 MiB) [ 3.888709] vda: detected capacity change from 0 to 71311360 [ 3.906574] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.909308] vdb: detected capacity change from 0 to 1073741824 [ 3.923853] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.926509] vdc: detected capacity change from 0 to 2621440000 [ 3.940477] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.943684] vdd: detected capacity change from 0 to 2621440000 [ 3.957338] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.960173] vde: detected capacity change from 0 to 4294967296 [ 3.973425] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.976413] vdf: detected capacity change from 0 to 4294967296 [ 3.983386] libphy: Fixed MDIO Bus: probed [ 3.999507] usbcore: registered new interface driver usbserial_generic [ 4.003222] usbserial: USB Serial support registered for generic [ 4.005737] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.009885] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.011736] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.014619] mousedev: PS/2 mouse device common for all mice [ 4.018261] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.019474] rtc_cmos 00:05: RTC can wake from S4 [ 4.035853] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.036616] rtc_cmos 00:05: registered as rtc0 [ 4.043265] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.045878] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.047750] intel_pstate: CPU model not supported [ 4.055723] hid: raw HID events driver (C) Jiri Kosina [ 4.057580] usbcore: registered new interface driver usbhid [ 4.059583] usbhid: USB HID core driver [ 4.061027] drop_monitor: Initializing network drop monitor service [ 4.063345] Initializing XFRM netlink socket [ 4.065374] NET: Registered protocol family 10 [ 4.068353] Segment Routing with IPv6 [ 4.069483] NET: Registered protocol family 17 [ 4.071519] mpls_gso: MPLS GSO support [ 4.076885] RAS: Correctable Errors collector initialized. [ 4.078719] AVX version of gcm_enc/dec engaged. [ 4.080196] AES CTR mode by8 optimization enabled [ 4.155514] sched_clock: Marking stable (4155482258, 0)->(5051018862, -895536604) [ 4.159330] registered taskstats version 1 [ 4.161349] Loading compiled-in X.509 certificates [ 4.163501] zswap: loaded using pool lzo/zbud [ 4.188325] Key type big_key registered [ 4.200742] Key type encrypted registered [ 4.202784] ima: No TPM chip found, activating TPM-bypass! [ 4.205259] ima: Allocated hash algorithm: sha1 [ 4.207400] ima: No architecture policies found [ 4.208967] evm: Initialising EVM extended attributes: [ 4.210739] evm: security.selinux [ 4.211693] evm: security.ima [ 4.212553] evm: security.capability [ 4.213528] evm: HMAC attrs: 0x1 [ 4.215590] rtc_cmos 00:05: setting system clock to 2026-06-17 01:42:23 UTC (1781660543) [ 4.221797] debug: unmapping init [mem 0xffffffff9d203000-0xffffffff9d3fffff] [ 4.224388] debug: unmapping init [mem 0xffffffff9bf82000-0xffffffff9c258fff] [ 4.232156] Write protecting the kernel read-only data: 28672k [ 4.236551] debug: unmapping init [mem 0xffffffff9a603000-0xffffffff9a7fffff] [ 4.240402] debug: unmapping init [mem 0xffffffff9af14000-0xffffffff9affffff] [ 4.290891] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.300666] systemd[1]: Detected virtualization kvm. [ 4.302854] systemd[1]: Detected architecture x86-64. [ 4.304992] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.332527] systemd[1]: No hostname configured. [ 4.334945] systemd[1]: Set hostname to . [ 4.336981] random: systemd: uninitialized urandom read (16 bytes read) [ 4.339468] systemd[1]: Initializing machine ID from random generator. [ 4.535666] random: systemd: uninitialized urandom read (16 bytes read) [ 4.538837] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.543671] random: systemd: uninitialized urandom read (16 bytes read) [ 4.546915] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.551737] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.908213] device-mapper: uevent: version 1.0.3 [ 5.911900] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.940647] virtio_net virtio0 ens2: renamed from eth0 [ 6.950456] random: fast init done [ 7.119729] scsi host0: ata_piix [ 7.136689] scsi host1: ata_piix [ 7.140239] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.148451] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.720971] random: crng init done [ 12.722324] random: 7 urandom warning(s) missed due to ratelimiting [ 12.844448] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.197496] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.060861] printk: systemd: 21 output lines suppressed due to ratelimiting [ 16.603899] SELinux: Disabled at runtime. [ 16.718196] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.728705] systemd[1]: Detected virtualization kvm. [ 16.730793] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.515874] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.520650] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.528287] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.538611] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.542814] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.551375] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.562386] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 18.060253] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 18.780946] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.748339] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.913980] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.340722] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.524374] EDAC sbridge: Ver: 1.1.2 [ 23.335380] Key type dns_resolver registered [ 23.896408] NFS: Registering the id_resolver key type [ 23.898832] Key type id_resolver registered [ 23.902260] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg343-server login: [ 54.370804] libcfs: loading out-of-tree module taints kernel. [ 54.394896] Key type ._llcrypt registered [ 54.396478] Key type .llcrypt registered [ 54.447801] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 63.158212] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 63.744043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 63.750378] alg: No test for adler32 (adler32-zlib) [ 64.740876] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 65.027632] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 66.647121] Key type lgssc registered [ 67.183934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.180882] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 87.818524] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 92.176913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 92.184630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 93.283591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 93.296954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 93.335691] Lustre: lustre-MDT0000: new disk, initializing [ 93.361292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.370285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 94.748036] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 99.763645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 99.808983] Lustre: 6477:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 99.823672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 99.826161] Lustre: Skipped 1 previous similar message [ 99.866120] Lustre: lustre-MDT0001: new disk, initializing [ 99.885453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 99.894848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 99.897973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 101.276923] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.590975] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 106.742303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 106.830424] Lustre: lustre-OST0000: new disk, initializing [ 106.831930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 106.834126] Lustre: 8415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 106.854158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 108.727922] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.644336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 113.648638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 113.665197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 114.000049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 114.052069] Lustre: lustre-OST0001: new disk, initializing [ 114.054313] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 114.057791] Lustre: 9486:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 114.079727] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 116.192977] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 121.848061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 121.852139] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 121.869846] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 122.798333] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 128.754131] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 130.877322] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing check_logdir /tmp/testlogs/ [ 132.392354] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing yml_node [ 133.896609] Lustre: DEBUG MARKER: Client: 2.17.54.40 [ 134.749599] Lustre: DEBUG MARKER: MDS: 2.17.54.40 [ 135.576394] Lustre: DEBUG MARKER: OSS: 2.17.54.40 [ 136.147227] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 16 21:44:34 EDT 2026 [ 141.739951] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 142.266711] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 156.127393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 156.129636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 156.133555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 157.664436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 157.664521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 157.667191] Lustre: Skipped 1 previous similar message [ 157.672812] Lustre: Skipped 1 previous similar message [ 160.155734] Lustre: server umount lustre-MDT0000 complete [ 161.479624] LustreError: 9485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781660700 with bad export cookie 7446307668300307889 [ 161.481173] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 161.484133] LustreError: 9485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 162.783731] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 162.784096] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 162.789391] Lustre: Skipped 2 previous similar messages [ 162.791331] Lustre: Skipped 2 previous similar messages [ 166.815921] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 166.820492] Lustre: Skipped 1 previous similar message [ 167.661606] Lustre: server umount lustre-MDT0001 complete [ 175.171623] Lustre: server umount lustre-OST0000 complete [ 182.608098] Lustre: server umount lustre-OST0001 complete [ 184.910593] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 188.255741] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 201.451252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.554031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 201.567224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 201.613810] Lustre: lustre-MDT0000: new disk, initializing [ 201.637137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.644725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 203.006537] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 207.458675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.502902] Lustre: 15028:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 207.638802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.129894] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 213.163100] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 215.221345] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 217.655572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.755870] Lustre: lustre-OST0000: new disk, initializing [ 217.757682] Lustre: Skipped 1 previous similar message [ 217.759460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 217.761622] Lustre: Skipped 2 previous similar messages [ 217.763753] Lustre: 16995:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.795774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.798448] Lustre: Skipped 1 previous similar message [ 218.811861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 218.815360] Lustre: Skipped 1 previous similar message [ 218.818256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 218.851620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 219.785900] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 223.822837] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 225.451748] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 225.537244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 227.004473] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 227.081622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 229.345990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 229.347215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 229.350072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 229.354754] Lustre: Skipped 1 previous similar message [ 239.585858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 239.588636] Lustre: Skipped 4 previous similar messages [ 242.143085] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 242.203441] Lustre: server umount lustre-OST0000 complete [ 243.680707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 243.684197] LustreError: Skipped 1 previous similar message [ 243.686446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 249.754501] Lustre: server umount lustre-MDT0000 complete [ 249.824033] LustreError: 15040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 249.829484] LustreError: 15040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 250.992169] LustreError: 16994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781660790 with bad export cookie 7446307668300309422 [ 250.995713] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 250.996721] LustreError: 16994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 251.109696] Lustre: server umount lustre-MDT0001 complete [ 252.362634] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 21:46:31 (1781660791) [ 254.649143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 254.802056] LustreError: 19256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 254.827031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 256.199479] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 259.188074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 260.801912] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 263.360176] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 265.211789] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 267.436302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 269.538420] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 272.098551] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 290.274625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 290.275544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.278580] LustreError: Skipped 1 previous similar message [ 290.283319] Lustre: Skipped 1 previous similar message [ 290.284268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 290.291337] Lustre: Skipped 4 previous similar messages [ 291.534793] Lustre: server umount lustre-OST0000 complete [ 295.392664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 307.167128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 307.231675] Lustre: server umount lustre-MDT0000 complete [ 308.571765] LustreError: 19237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781660847 with bad export cookie 7446307668300310304 [ 308.575208] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 308.575285] LustreError: 19237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 308.697722] Lustre: server umount lustre-MDT0001 complete [ 312.780516] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 313.869100] Key type lgssc unregistered [ 314.005429] LNet: 22806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 314.008072] LNetError: 22806:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 314.014937] LNet: Removed LNI 192.168.203.143@tcp [ 314.367133] Key type .llcrypt unregistered [ 314.368784] Key type ._llcrypt unregistered [ 323.468107] Key type ._llcrypt registered [ 323.469163] Key type .llcrypt registered [ 323.504898] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 323.887454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 323.908670] alg: No test for adler32 (adler32-zlib) [ 324.758491] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 324.837733] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 326.415145] Key type lgssc registered [ 326.766515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 329.399518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 329.407825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 330.572478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 332.009515] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 335.019287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 335.183725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 336.652822] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 339.239348] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 341.165513] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 343.554162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 343.668724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 345.619547] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 348.140415] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 354.918993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 360.417856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 360.419076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.421377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 360.426354] Lustre: Skipped 1 previous similar message [ 360.430239] Lustre: Skipped 1 previous similar message [ 363.857214] Lustre: server umount lustre-OST0000 complete [ 365.536570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.542317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.544873] Lustre: Skipped 1 previous similar message [ 370.656036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 370.658894] Lustre: Skipped 1 previous similar message [ 371.282372] Lustre: server umount lustre-MDT0000 complete [ 372.657537] LustreError: 24174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781660911 with bad export cookie 6445402239686113682 [ 372.660923] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 372.663262] LustreError: 24174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 372.799835] Lustre: server umount lustre-MDT0001 complete [ 376.766798] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 377.948561] Key type lgssc unregistered [ 378.079531] LNet: 28167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.082581] LNetError: 28167:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 378.091389] LNet: Removed LNI 192.168.203.143@tcp [ 378.432402] Key type .llcrypt unregistered [ 378.434357] Key type ._llcrypt unregistered [ 387.446059] Key type ._llcrypt registered [ 387.447310] Key type .llcrypt registered [ 387.487165] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 387.880661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 387.889423] alg: No test for adler32 (adler32-zlib) [ 388.749630] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 388.833876] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 390.415181] Key type lgssc registered [ 390.839840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 393.825192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 393.834477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.026438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 396.618974] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 399.751730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.882376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 401.277067] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 403.840594] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 405.851938] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 410.722701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 410.884928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.092270] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 415.760070] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 421.158066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 430.564349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 430.565542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.568591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 430.571435] Lustre: Skipped 1 previous similar message [ 430.576488] Lustre: Skipped 1 previous similar message [ 434.570631] Lustre: server umount lustre-OST0000 complete [ 440.800798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.800966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 440.801580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.804816] Lustre: Skipped 1 previous similar message [ 441.946568] Lustre: server umount lustre-MDT0000 complete [ 443.247429] LustreError: 29538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781660982 with bad export cookie 11655352550265812325 [ 443.250925] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 443.255169] LustreError: 29538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 443.368636] Lustre: server umount lustre-MDT0001 complete [ 447.109236] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 448.176848] Key type lgssc unregistered [ 448.307814] LNet: 33453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 448.311637] LNetError: 33453:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 448.320365] LNet: Removed LNI 192.168.203.143@tcp [ 448.616800] Key type .llcrypt unregistered [ 448.618104] Key type ._llcrypt unregistered [ 457.755984] Key type ._llcrypt registered [ 457.757422] Key type .llcrypt registered [ 457.797743] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 458.100629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 458.111653] alg: No test for adler32 (adler32-zlib) [ 458.970991] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 459.055224] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 460.639164] Key type lgssc registered [ 461.022816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 463.824805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 463.832424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 464.996191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.464084] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 469.464747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 469.591138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 470.985383] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 473.499460] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 475.393753] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 477.743362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 477.872806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.938718] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 482.505720] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 490.150326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 525.280340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 525.284068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.289364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 526.305704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 526.306099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 526.306397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.315793] Lustre: Skipped 1 previous similar message [ 528.076859] Lustre: server umount lustre-OST0000 complete [ 530.911877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 530.914279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.920280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 535.509108] Lustre: server umount lustre-MDT0000 complete [ 536.543858] LustreError: 36245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.548920] LustreError: 36245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 536.767235] LustreError: 35577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661076 with bad export cookie 17223438386190110162 [ 536.769854] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 536.771786] LustreError: 35577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 536.884180] Lustre: server umount lustre-MDT0001 complete [ 540.553275] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 541.681722] Key type lgssc unregistered [ 541.821332] LNet: 38398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.824651] LNetError: 38398:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 541.833275] LNet: Removed LNI 192.168.203.143@tcp [ 542.141719] Key type .llcrypt unregistered [ 542.143732] Key type ._llcrypt unregistered [ 551.671300] Key type ._llcrypt registered [ 551.672795] Key type .llcrypt registered [ 551.715081] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 552.144047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 552.165197] alg: No test for adler32 (adler32-zlib) [ 553.046119] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 553.134561] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 554.727128] Key type lgssc registered [ 555.117539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 558.036693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 558.045690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.229468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 560.784558] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 563.931692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.070231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 565.527963] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 568.160730] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 570.081653] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 572.535317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 572.665850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.767282] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 577.405516] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 583.913994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 590.304058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 590.306730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.312423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 593.100208] Lustre: lustre-OST0000: Not available for connect from 192.168.203.43@tcp (stopping) [ 594.403810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 594.403899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 594.404365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.417175] Lustre: Skipped 1 previous similar message [ 596.371836] Lustre: server umount lustre-OST0000 complete [ 609.589745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.590667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 609.595027] Lustre: Skipped 1 previous similar message [ 609.600470] Lustre: Skipped 1 previous similar message [ 614.881077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 614.885219] Lustre: Skipped 1 previous similar message [ 615.770895] Lustre: server umount lustre-MDT0000 complete [ 617.236948] LustreError: 39767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661156 with bad export cookie 8435956749352544671 [ 617.239453] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 617.244622] LustreError: 39767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 623.416752] Lustre: server umount lustre-MDT0001 complete [ 627.436934] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 628.584796] Key type lgssc unregistered [ 628.730387] LNet: 43514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.733961] LNetError: 43514:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.745525] LNet: Removed LNI 192.168.203.143@tcp [ 629.060339] Key type .llcrypt unregistered [ 629.061833] Key type ._llcrypt unregistered [ 638.796790] Key type ._llcrypt registered [ 638.798196] Key type .llcrypt registered [ 638.837214] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 639.293539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 639.312377] alg: No test for adler32 (adler32-zlib) [ 640.184200] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 640.274119] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 641.863165] Key type lgssc registered [ 642.319730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 645.331564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 645.339286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.529817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 648.033586] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 651.267944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.421450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 652.907441] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 655.524430] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 657.617343] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 660.127722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 660.273693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.441194] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 665.084655] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 671.530806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 680.652801] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.43@tcp (stopping) [ 681.952807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 681.954704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 681.959172] Lustre: Skipped 2 previous similar messages [ 681.962613] Lustre: Skipped 1 previous similar message [ 684.119856] Lustre: server umount lustre-MDT0000 complete [ 685.504273] LustreError: 44885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661224 with bad export cookie 12554719117468330462 [ 685.507796] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 685.510489] LustreError: 44885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 685.763229] LustreError: 46890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 685.770727] LustreError: 46890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 685.770821] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.43@tcp (stopping) [ 687.071888] LustreError: 44897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 687.072234] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.079831] LustreError: 46890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.080117] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 690.089389] Lustre: Skipped 1 previous similar message [ 691.700274] Lustre: server umount lustre-MDT0001 complete [ 709.656647] Lustre: server umount lustre-OST0000 complete [ 715.638858] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 716.723756] Key type lgssc unregistered [ 716.868298] LNet: 48781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.871869] LNetError: 48781:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.879304] LNet: Removed LNI 192.168.203.143@tcp [ 717.194507] Key type .llcrypt unregistered [ 717.196163] Key type ._llcrypt unregistered [ 727.019586] Key type ._llcrypt registered [ 727.021020] Key type .llcrypt registered [ 727.057162] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 727.459488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 727.468367] alg: No test for adler32 (adler32-zlib) [ 728.322572] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 728.406133] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 729.983165] Key type lgssc registered [ 730.326327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 732.930175] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 732.937726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 748.383313] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 753.503199] LustreError: 50144:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 780.127166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 781.996547] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 784.305398] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 949.238012] Lustre: server umount lustre-OST0000 complete [ 954.223362] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 955.157357] Key type lgssc unregistered [ 955.280286] LNet: 51815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.282455] LNetError: 51815:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.290239] LNet: Removed LNI 192.168.203.143@tcp [ 955.570215] Key type .llcrypt unregistered [ 955.571730] Key type ._llcrypt unregistered [ 963.708066] Key type ._llcrypt registered [ 963.709104] Key type .llcrypt registered [ 963.738090] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 964.050045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 964.072460] alg: No test for adler32 (adler32-zlib) [ 964.927964] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 965.013118] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 966.599175] Key type lgssc registered [ 966.972309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 969.388667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 969.393698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 970.534259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 971.771543] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 974.496622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.609621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 975.844249] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 978.065369] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 979.674155] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 981.848401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 981.957985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.807328] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 986.223409] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 989.153348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.154052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 989.157069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 993.037896] Lustre: server umount lustre-OST0000 complete [ 994.273788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.275024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.277267] Lustre: Skipped 1 previous similar message [ 994.282420] Lustre: Skipped 1 previous similar message [ 999.392236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 999.394725] Lustre: Skipped 1 previous similar message [ 1000.408774] Lustre: server umount lustre-MDT0000 complete [ 1001.642252] LustreError: 53185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661540 with bad export cookie 9915932892426873391 [ 1001.647142] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1001.647436] LustreError: 53185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1001.781861] Lustre: server umount lustre-MDT0001 complete [ 1005.390083] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1006.427103] Key type lgssc unregistered [ 1006.552717] LNet: 56733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.555463] LNetError: 56733:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.564373] LNet: Removed LNI 192.168.203.143@tcp [ 1006.874661] Key type .llcrypt unregistered [ 1006.875763] Key type ._llcrypt unregistered [ 1015.807735] Key type ._llcrypt registered [ 1015.808668] Key type .llcrypt registered [ 1015.846635] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1016.273930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1016.282471] alg: No test for adler32 (adler32-zlib) [ 1017.137685] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1017.224821] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1018.807188] Key type lgssc registered [ 1019.178906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1021.783876] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1021.792538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1037.215345] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1042.335200] LustreError: 58097:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 1068.959156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.701800] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1072.746069] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1075.046713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1075.218850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.241700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1076.595423] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1079.497333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1079.645983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1081.057052] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1083.440215] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1085.303790] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1088.993651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1088.994500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1088.996691] Lustre: Skipped 1 previous similar message [ 1094.113953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1094.115733] Lustre: Skipped 2 previous similar messages [ 1099.233466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1099.236274] Lustre: Skipped 1 previous similar message [ 1100.255138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1100.306508] Lustre: server umount lustre-OST0000 complete [ 1115.448657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.449450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1115.452455] Lustre: Skipped 1 previous similar message [ 1120.736069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1120.738273] Lustre: Skipped 2 previous similar messages [ 1121.684549] Lustre: server umount lustre-MDT0000 complete [ 1122.980656] LustreError: 58103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661662 with bad export cookie 9162308833591023666 [ 1122.982770] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1122.985300] LustreError: 58103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1129.136575] Lustre: server umount lustre-MDT0001 complete [ 1132.833980] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1133.902391] Key type lgssc unregistered [ 1134.031328] LNet: 61854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.034676] LNetError: 61854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.043280] LNet: Removed LNI 192.168.203.143@tcp [ 1134.335839] Key type .llcrypt unregistered [ 1134.337036] Key type ._llcrypt unregistered [ 1143.344633] Key type ._llcrypt registered [ 1143.345624] Key type .llcrypt registered [ 1143.382354] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1143.729121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1143.781968] alg: No test for adler32 (adler32-zlib) [ 1144.634879] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1144.722360] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1146.295127] Key type lgssc registered [ 1146.644396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1149.152093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1149.158034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.302580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1151.562477] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1154.305938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.435041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1155.725299] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1158.003495] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1159.819535] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1162.074336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1162.195143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.152961] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1166.512968] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1175.526066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1179.615860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1179.618037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.621568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1180.641822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1180.642052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.642252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1184.974275] Lustre: server umount lustre-OST0000 complete [ 1190.367862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1190.371317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.377563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1190.379901] Lustre: Skipped 1 previous similar message [ 1192.339528] Lustre: server umount lustre-MDT0000 complete [ 1193.523096] LustreError: 63222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661732 with bad export cookie 13204499701696805856 [ 1193.526077] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1193.529513] LustreError: 63222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1193.648554] Lustre: server umount lustre-MDT0001 complete [ 1197.175362] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1198.158972] Key type lgssc unregistered [ 1198.287347] LNet: 66791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.289632] LNetError: 66791:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1198.298292] LNet: Removed LNI 192.168.203.143@tcp [ 1198.582304] Key type .llcrypt unregistered [ 1198.583332] Key type ._llcrypt unregistered [ 1207.971814] Key type ._llcrypt registered [ 1207.972738] Key type .llcrypt registered [ 1208.006392] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1208.339651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1208.392092] alg: No test for adler32 (adler32-zlib) [ 1209.236415] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1209.313764] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1210.895182] Key type lgssc registered [ 1211.254881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1214.215769] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 22:02:32 (1781661752) [ 1216.430457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1216.436744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.592143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.970339] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1221.776468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.897133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1223.235535] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1225.706372] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1227.538451] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1229.864533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1229.979577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.955724] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1234.386644] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1242.284189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1247.017161] Lustre: Failing over lustre-MDT0000 [ 1247.208958] Lustre: server umount lustre-MDT0000 complete [ 1247.713120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1247.716072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.720881] LustreError: 68472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.727156] LustreError: 68472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1249.994740] LustreError: 68466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.383038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.421826] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1250.474067] Lustre: *** cfs_fail_loc=135, val=0*** [ 1250.516023] LustreError: 71576:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1250.518161] LustreError: 71576:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.143@tcp: cfg command failed: rc = -2 [ 1250.520739] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1250.520739] [ 1250.523648] LustreError: MGC192.168.203.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1250.530343] LustreError: 71564:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1250.534555] LustreError: 71564:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1250.539301] LustreError: 71564:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1250.542429] LustreError: 71564:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1250.552594] Lustre: server umount lustre-MDT0000 complete [ 1250.554162] LustreError: 71564:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1252.724650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1252.771302] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1252.857503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1252.887574] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1254.191509] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1255.107157] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1257.960606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1257.969066] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1257.987496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1263.073017] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1263.074858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.075887] LustreError: Skipped 1 previous similar message [ 1263.080334] Lustre: Skipped 2 previous similar messages [ 1263.080938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1263.085450] Lustre: Skipped 1 previous similar message [ 1265.099704] Lustre: server umount lustre-OST0000 complete [ 1268.192937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.193033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1268.193833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1268.196608] Lustre: Skipped 2 previous similar messages [ 1272.472472] Lustre: server umount lustre-MDT0000 complete [ 1273.312978] LustreError: 68472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.319229] LustreError: 68472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1273.731201] LustreError: 70312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661813 with bad export cookie 12571504543157109512 [ 1273.734387] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1273.737923] LustreError: 70312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1273.856615] Lustre: server umount lustre-MDT0001 complete [ 1277.431242] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1278.443809] Key type lgssc unregistered [ 1278.572486] LNet: 73350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.575200] LNetError: 73350:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1278.581268] LNet: Removed LNI 192.168.203.143@tcp [ 1278.908664] Key type .llcrypt unregistered [ 1278.910035] Key type ._llcrypt unregistered [ 1287.556781] Key type ._llcrypt registered [ 1287.558691] Key type .llcrypt registered [ 1287.595019] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1287.875414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1287.887492] alg: No test for adler32 (adler32-zlib) [ 1288.737279] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1288.813918] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1290.391101] Key type lgssc registered [ 1290.760861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1293.199480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1293.206296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1294.358223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.703926] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1298.515668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1298.629892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1299.944584] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1302.273124] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1303.994467] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1306.180984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1306.292440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1308.159586] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1310.436093] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1319.589467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1323.291360] Lustre: Failing over lustre-MDT0000 [ 1323.458086] Lustre: server umount lustre-MDT0000 complete [ 1324.511758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1324.513874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1324.518674] LustreError: 74740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.524898] LustreError: 74740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1325.024711] LustreError: 76180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1325.025846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.283098] LustreError: 77231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1326.288016] LustreError: 77231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1326.517722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1326.558968] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1326.616269] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1326.675552] LustreError: 77849:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1326.680151] LustreError: 77849:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.143@tcp: cfg command failed: rc = -9 [ 1326.685689] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1326.685689] [ 1326.691402] LustreError: MGC192.168.203.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1326.695936] LustreError: 77838:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1326.698350] LustreError: 77838:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 1326.701249] LustreError: 77838:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1326.704347] LustreError: 77838:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1326.717669] Lustre: server umount lustre-MDT0000 complete [ 1326.720215] LustreError: 77838:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1328.904889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.947509] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1329.041590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1329.066568] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1330.423659] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1331.395315] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1334.249534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1334.257140] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1334.277092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1339.363486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1339.364309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.366017] LustreError: Skipped 1 previous similar message [ 1339.371248] Lustre: Skipped 1 previous similar message [ 1339.373884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1339.377150] Lustre: Skipped 1 previous similar message [ 1341.326886] Lustre: server umount lustre-OST0000 complete [ 1344.481385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1344.481384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1344.482242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1344.490979] Lustre: Skipped 1 previous similar message [ 1348.762819] Lustre: server umount lustre-MDT0000 complete [ 1349.599973] LustreError: 74740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.605928] LustreError: 74740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1349.960237] LustreError: 74719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661889 with bad export cookie 9668682990666845521 [ 1349.963104] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1349.964076] LustreError: 74719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1350.076797] Lustre: server umount lustre-MDT0001 complete [ 1353.647140] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1354.677843] Key type lgssc unregistered [ 1354.809344] LNet: 79623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1354.812168] LNetError: 79623:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1354.820317] LNet: Removed LNI 192.168.203.143@tcp [ 1355.101229] Key type .llcrypt unregistered [ 1355.102664] Key type ._llcrypt unregistered [ 1359.107590] Key type ._llcrypt registered [ 1359.108420] Key type .llcrypt registered [ 1359.140762] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1359.226339] Key type .llcrypt unregistered [ 1359.227218] Key type ._llcrypt unregistered [ 1367.165877] Key type ._llcrypt registered [ 1367.166945] Key type .llcrypt registered [ 1367.200556] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1367.526085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1367.550710] alg: No test for adler32 (adler32-zlib) [ 1368.409961] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1368.499820] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1370.079163] Key type lgssc registered [ 1370.429605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1373.082443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1373.088726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.242246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.552618] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1378.371671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1378.503914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1379.864711] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1382.234678] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1383.956091] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1386.230298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1386.343532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.355684] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1390.855430] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1400.679205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1405.922075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.923324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1405.925354] Lustre: Skipped 1 previous similar message [ 1405.925685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1405.926850] Lustre: Skipped 1 previous similar message [ 1410.251238] Lustre: server umount lustre-OST0000 complete [ 1414.623958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1414.627355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1414.633851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1416.160297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1416.160879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.624780] Lustre: server umount lustre-MDT0000 complete [ 1418.808240] LustreError: 81479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661958 with bad export cookie 10591023610521359237 [ 1418.811435] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1418.812157] LustreError: 81479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1418.934455] Lustre: server umount lustre-MDT0001 complete [ 1422.479665] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1423.512225] Key type lgssc unregistered [ 1423.640345] LNet: 85052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.642861] LNetError: 85052:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1423.652301] LNet: Removed LNI 192.168.203.143@tcp [ 1423.943535] Key type .llcrypt unregistered [ 1423.945728] Key type ._llcrypt unregistered [ 1432.362032] Key type ._llcrypt registered [ 1432.362958] Key type .llcrypt registered [ 1432.397647] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1432.757713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1432.833993] alg: No test for adler32 (adler32-zlib) [ 1433.689366] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1433.773420] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1435.359168] Key type lgssc registered [ 1435.697586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1438.169648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1438.175054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1439.326821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1440.604982] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1443.346065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.474775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1444.759562] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1447.015974] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1448.810366] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1451.047823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1451.182363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.048030] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1455.313670] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1465.511563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1468.896310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1468.899741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1468.905390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1470.946952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1470.947416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1470.947507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1474.382920] Lustre: server umount lustre-OST0000 complete [ 1476.064843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.065647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1476.068682] Lustre: Skipped 1 previous similar message [ 1476.072085] Lustre: Skipped 2 previous similar messages [ 1481.184216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.186621] Lustre: Skipped 1 previous similar message [ 1481.818912] Lustre: server umount lustre-MDT0000 complete [ 1483.038945] LustreError: 86420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662022 with bad export cookie 1455735657675943533 [ 1483.042381] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1483.042594] LustreError: 86420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1483.170417] Lustre: server umount lustre-MDT0001 complete [ 1486.890135] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1487.895652] Key type lgssc unregistered [ 1488.010350] LNet: 89989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.012813] LNetError: 89989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1488.024326] LNet: Removed LNI 192.168.203.143@tcp [ 1488.330270] Key type .llcrypt unregistered [ 1488.331225] Key type ._llcrypt unregistered [ 1497.138529] Key type ._llcrypt registered [ 1497.139592] Key type .llcrypt registered [ 1497.176572] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1497.576223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1497.592457] alg: No test for adler32 (adler32-zlib) [ 1498.449749] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1498.526900] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1500.103125] Key type lgssc registered [ 1500.473682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1503.006894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1503.013066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1504.165482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1505.513217] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1508.306408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.427383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1509.722282] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1511.987428] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1513.812687] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1516.063262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1516.174292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1518.147407] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1520.582262] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1529.511069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1534.945916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1534.948194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.957123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1534.960537] Lustre: Skipped 1 previous similar message [ 1540.064642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1540.361291] Lustre: server umount lustre-OST0000 complete [ 1544.159841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1544.162694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.167052] Lustre: Skipped 1 previous similar message [ 1544.168792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1544.170993] Lustre: Skipped 1 previous similar message [ 1545.185121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.666571] Lustre: server umount lustre-MDT0000 complete [ 1548.873257] LustreError: 91359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662088 with bad export cookie 10285523671147900313 [ 1548.876143] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1548.877693] LustreError: 91359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1548.984912] Lustre: server umount lustre-MDT0001 complete [ 1552.545722] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1553.520947] Key type lgssc unregistered [ 1553.638373] LNet: 94930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.640800] LNetError: 94930:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1553.648261] LNet: Removed LNI 192.168.203.143@tcp [ 1553.930226] Key type .llcrypt unregistered [ 1553.931980] Key type ._llcrypt unregistered [ 1562.653209] Key type ._llcrypt registered [ 1562.654115] Key type .llcrypt registered [ 1562.692527] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1563.002708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1563.066178] alg: No test for adler32 (adler32-zlib) [ 1563.922782] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1563.999410] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1565.583158] Key type lgssc registered [ 1565.941068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1568.512480] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1568.520686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1583.967280] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1589.087166] LustreError: 96293:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 1615.711165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.528496] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1619.795804] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1621.568558] Lustre: server umount lustre-OST0000 complete [ 1624.167245] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 22:09:22 (1781662162) [ 1626.337297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.480316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1627.804239] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1630.581185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.704405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1632.078250] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1634.335624] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1636.044569] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1638.206664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1638.323530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1640.233854] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1642.524086] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1651.622317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1656.800420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1656.801392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.802240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.810142] Lustre: Skipped 1 previous similar message [ 1658.254832] Lustre: server umount lustre-OST0000 complete [ 1661.407970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1661.412676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.420815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1661.422687] Lustre: Skipped 1 previous similar message [ 1661.525891] Lustre: server umount lustre-MDT0000 complete [ 1661.920694] LustreError: 97923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1661.927904] LustreError: 97923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1662.708218] LustreError: 97911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662201 with bad export cookie 14600550117090941352 [ 1662.711692] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1662.713670] LustreError: 97911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1662.827079] Lustre: server umount lustre-MDT0001 complete [ 1666.171320] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1667.242157] Key type lgssc unregistered [ 1667.381437] LNet: 101637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.384098] LNetError: 101637:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.392466] LNet: Removed LNI 192.168.203.143@tcp [ 1667.724596] Key type .llcrypt unregistered [ 1667.725732] Key type ._llcrypt unregistered [ 1681.838393] Key type ._llcrypt registered [ 1681.839520] Key type .llcrypt registered [ 1681.876214] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1682.284131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1682.290180] alg: No test for adler32 (adler32-zlib) [ 1683.134927] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1683.208879] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1684.783156] Key type lgssc registered [ 1685.140236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1687.616789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1687.626316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1688.788483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1690.033297] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1692.717582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.853105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1694.096627] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1696.350628] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1698.068749] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1700.345019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1700.451480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.278300] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1704.599546] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1713.764951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1719.264910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1719.266461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1719.271921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1719.276706] Lustre: Skipped 1 previous similar message [ 1719.280869] Lustre: Skipped 1 previous similar message [ 1723.529849] Lustre: server umount lustre-OST0000 complete [ 1728.992085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1728.996637] LustreError: Skipped 1 previous similar message [ 1728.999162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.006900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.624501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.628758] Lustre: Skipped 3 previous similar messages [ 1738.720396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1738.722626] Lustre: Skipped 1 previous similar message [ 1739.231109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1739.284038] Lustre: server umount lustre-MDT0000 complete [ 1740.502137] LustreError: 103778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662279 with bad export cookie 4698100986619891524 [ 1740.506121] LustreError: 103778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1740.506779] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1740.632342] Lustre: server umount lustre-MDT0001 complete [ 1744.051521] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1744.977862] Key type lgssc unregistered [ 1745.097361] LNet: 107346:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.101055] LNetError: 107346:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.108324] LNet: Removed LNI 192.168.203.143@tcp [ 1745.401244] Key type .llcrypt unregistered [ 1745.402245] Key type ._llcrypt unregistered [ 1752.418910] Key type ._llcrypt registered [ 1752.419997] Key type .llcrypt registered [ 1752.456292] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1752.761972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1752.834212] alg: No test for adler32 (adler32-zlib) [ 1753.682232] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1753.763675] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1755.343137] Key type lgssc registered [ 1755.671896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1758.143693] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1758.149291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1773.599229] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1778.719131] LustreError: 108566:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 1805.343137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.114009] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1809.502951] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1811.834405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.929793] LustreError: MGC192.168.203.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1811.934798] LustreError: 109590:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1811.937383] LustreError: 109590:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1811.941763] LustreError: 109590:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1811.973267] Lustre: server umount lustre-MDT0000 complete [ 1811.975254] LustreError: 109590:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1824.767808] Lustre: server umount lustre-OST0000 complete [ 1827.821216] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 1830.981105] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1843.880939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.993620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1844.005145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1844.048399] Lustre: lustre-MDT0000: new disk, initializing [ 1844.072499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1844.080488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1845.415607] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1849.693248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.748910] Lustre: 112376:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1849.776903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1849.779840] Lustre: Skipped 1 previous similar message [ 1849.818570] Lustre: lustre-MDT0001: new disk, initializing [ 1849.850006] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1849.861261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1849.864531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1851.150566] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1854.711133] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1856.434875] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1858.707547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1858.821795] Lustre: lustre-OST0000: new disk, initializing [ 1858.824061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1858.826783] Lustre: 114343:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1858.854434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1859.774962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1859.780444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1859.830584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1860.902607] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1864.834603] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1866.207315] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1866.274161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1867.706165] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1867.777150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1870.306679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.307420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1870.313394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1870.314731] LustreError: Skipped 1 previous similar message [ 1870.317830] Lustre: Skipped 1 previous similar message [ 1870.540413] Lustre: server umount lustre-OST0000 complete [ 1875.426355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.426574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1875.434807] Lustre: Skipped 1 previous similar message [ 1875.444147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1875.448232] Lustre: Skipped 1 previous similar message [ 1880.545287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1880.550655] Lustre: Skipped 1 previous similar message [ 1885.664476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1885.667556] Lustre: Skipped 1 previous similar message [ 1886.175175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1886.260607] Lustre: server umount lustre-MDT0000 complete [ 1887.676242] LustreError: 112369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662426 with bad export cookie 15153671531206679583 [ 1887.682399] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1887.813739] Lustre: server umount lustre-MDT0001 complete [ 1890.860168] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 22:13:49 (1781662429) [ 1891.952461] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1897.187773] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 1900.188786] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1912.529597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.648023] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1912.664343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1912.705857] Lustre: lustre-MDT0000: new disk, initializing [ 1912.733232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.741975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1914.384368] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1919.061349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1919.129329] Lustre: 119232:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1919.160260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1919.163556] Lustre: Skipped 1 previous similar message [ 1919.208703] Lustre: lustre-MDT0001: new disk, initializing [ 1919.256132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1919.260349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1920.631673] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1924.682501] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1926.489673] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1928.853533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1928.957363] Lustre: lustre-OST0000: new disk, initializing [ 1928.959247] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1928.961983] Lustre: 121198:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1928.991559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.993735] Lustre: Skipped 1 previous similar message [ 1930.917632] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1934.795613] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1936.501365] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1938.439906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1938.447767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1938.497359] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1938.646304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1940.323291] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1940.445696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1943.521652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1943.522999] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.524106] LustreError: Skipped 1 previous similar message [ 1943.528781] Lustre: Skipped 1 previous similar message [ 1943.529605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1943.533940] Lustre: Skipped 1 previous similar message [ 1947.402385] Lustre: server umount lustre-OST0000 complete [ 1948.640181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.645226] Lustre: Skipped 2 previous similar messages [ 1953.760982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.764727] Lustre: Skipped 3 previous similar messages [ 1954.680164] Lustre: server umount lustre-MDT0000 complete [ 1956.124565] LustreError: 119224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662495 with bad export cookie 15153671531206680451 [ 1956.130484] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1956.133028] LustreError: 119224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1956.290644] Lustre: server umount lustre-MDT0001 complete [ 1959.833710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.106913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.776165] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1965.066561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.704116] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1969.055107] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1970.803846] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1972.817566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1974.618227] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1976.757927] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1990.005839] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1995.745594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.746647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1995.749874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1995.751169] LustreError: Skipped 1 previous similar message [ 1995.753860] Lustre: Skipped 1 previous similar message [ 1996.999317] Lustre: server umount lustre-OST0000 complete [ 2004.244528] Lustre: server umount lustre-MDT0000 complete [ 2005.721628] LustreError: 123408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662545 with bad export cookie 15153671531206681319 [ 2005.727554] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2005.730224] LustreError: 123408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2005.892687] Lustre: server umount lustre-MDT0001 complete [ 2010.048545] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2011.317405] Key type lgssc unregistered [ 2011.444576] LNet: 127073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.450354] LNetError: 127073:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.461406] LNet: Removed LNI 192.168.203.143@tcp [ 2011.885706] Key type .llcrypt unregistered [ 2011.886723] Key type ._llcrypt unregistered [ 2017.964296] Key type ._llcrypt registered [ 2017.965261] Key type .llcrypt registered [ 2018.005702] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 2023.579835] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2024.077368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2024.084808] alg: No test for adler32 (adler32-zlib) [ 2024.953391] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 2025.040356] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2026.631104] Key type lgssc registered [ 2027.058459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2040.320118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2040.330342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.430914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2041.440586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2041.475430] Lustre: lustre-MDT0000: new disk, initializing [ 2041.496874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.502926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2042.914395] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2047.186746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.227200] Lustre: 130197:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2047.252839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2047.254711] Lustre: Skipped 1 previous similar message [ 2047.293125] Lustre: lustre-MDT0001: new disk, initializing [ 2047.327716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2047.341897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2047.345190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2048.635392] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2052.287601] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2053.998688] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2056.240492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2056.341156] Lustre: lustre-OST0000: new disk, initializing [ 2056.342837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2056.345341] Lustre: 132165:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2056.373984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.529802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2056.532625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2056.563199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2058.415985] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2062.176124] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2063.498987] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2063.564069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2065.121811] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2065.189796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2066.916206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2066.916797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.920915] LustreError: Skipped 1 previous similar message [ 2066.934956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2066.942062] Lustre: Skipped 1 previous similar message [ 2072.033790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2072.035477] Lustre: Skipped 1 previous similar message [ 2077.154553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2077.156608] Lustre: Skipped 1 previous similar message [ 2080.223119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2080.273161] Lustre: server umount lustre-OST0000 complete [ 2082.272678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2082.273534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2082.277384] Lustre: Skipped 2 previous similar messages [ 2082.281838] Lustre: Skipped 1 previous similar message [ 2087.392488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2087.395179] Lustre: Skipped 1 previous similar message [ 2087.656154] Lustre: server umount lustre-MDT0000 complete [ 2088.892303] LustreError: 133704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662628 with bad export cookie 7504650629074081373 [ 2088.895968] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2088.896211] LustreError: 133704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2089.025941] Lustre: server umount lustre-MDT0001 complete [ 2091.609364] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 22:17:10 (1781662630) [ 2093.720064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.888798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.149552] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2097.850675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.970366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2099.271562] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2101.662593] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2103.362259] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2108.130137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.143437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2108.149951] Lustre: Skipped 1 previous similar message [ 2110.168810] Lustre: server umount lustre-MDT0000 complete [ 2111.322331] LustreError: 136141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662650 with bad export cookie 7504650629074082241 [ 2111.325354] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.328761] LustreError: 136141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2111.443985] Lustre: server umount lustre-MDT0001 complete [ 2113.952148] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 22:17:32 (1781662652) [ 2116.048796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2131.487720] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2136.607351] LustreError: 136985:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 2162.207535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2164.226201] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2166.680902] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2167.418937] Lustre: server umount lustre-OST0000 complete [ 2170.027163] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 22:18:28 (1781662708) [ 2172.156185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2172.301274] LustreError: 138427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.307973] LustreError: 138427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2173.574752] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2176.306072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.728061] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2180.019370] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2181.708733] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2183.847918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2183.953730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2183.955986] Lustre: Skipped 2 previous similar messages [ 2186.151111] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2188.553563] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2202.079971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2202.083418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.087907] Lustre: Skipped 1 previous similar message [ 2202.090287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2207.008298] Lustre: server umount lustre-MDT0000 complete [ 2207.712627] LustreError: 138428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.718183] LustreError: 138428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2208.318503] LustreError: 138409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662747 with bad export cookie 7504650629074082892 [ 2208.323152] LustreError: 138409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2208.324587] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2208.432868] Lustre: server umount lustre-MDT0001 complete [ 2220.078668] Lustre: server umount lustre-OST0000 complete [ 2223.371643] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 22:19:22 (1781662762) [ 2225.613074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2225.776027] LustreError: 142096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.800441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.151223] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2229.845097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2231.238418] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2233.539676] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2235.352042] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2237.514828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2239.484183] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2241.822244] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2243.274086] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2250.917530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2251.496664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2252.815822] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2252.889222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2256.352607] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2256.354704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2256.354847] LustreError: Skipped 1 previous similar message [ 2256.358298] Lustre: Skipped 2 previous similar messages [ 2256.359038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2256.362964] Lustre: Skipped 3 previous similar messages [ 2259.594237] Lustre: server umount lustre-OST0000 complete [ 2260.959786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2275.295063] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2275.360916] Lustre: server umount lustre-MDT0000 complete [ 2275.808230] LustreError: 144837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.813467] LustreError: 144837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2276.608384] LustreError: 142078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662815 with bad export cookie 7504650629074083949 [ 2276.611749] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2276.613034] LustreError: 142078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2276.723501] Lustre: server umount lustre-MDT0001 complete [ 2279.345679] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 22:20:18 (1781662818) [ 2281.684804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2297.119567] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2302.239361] LustreError: 146311:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 2327.839202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.841176] Lustre: Skipped 2 previous similar messages [ 2329.699611] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2331.973242] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2334.102594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2334.234811] LustreError: 146336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.279070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2335.583798] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2338.275714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.657167] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2342.077583] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2343.772288] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2345.084795] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2345.151488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2346.516848] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2346.585525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2348.000658] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.002227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2348.004692] Lustre: Skipped 4 previous similar messages [ 2348.006716] Lustre: Skipped 7 previous similar messages [ 2353.362576] Lustre: server umount lustre-MDT0000 complete [ 2354.653924] LustreError: 146318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662893 with bad export cookie 7504650629074084957 [ 2354.657358] LustreError: 146318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2354.657851] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2354.763083] Lustre: server umount lustre-MDT0001 complete [ 2366.422117] Lustre: server umount lustre-OST0000 complete [ 2368.969804] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 22:21:47 (1781662907) [ 2371.236232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2386.719676] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2391.839423] LustreError: 150501:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 2419.284276] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2421.530201] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2423.540248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.676245] LustreError: 150526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.716808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2424.954703] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2428.096780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2429.555193] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2432.041767] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2433.851147] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2436.506778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2436.587426] Lustre: lustre-OST0001: new disk, initializing [ 2436.590144] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2436.595817] Lustre: 153372:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2437.448764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2437.457064] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2437.501111] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2438.751735] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2442.413205] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2443.761518] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2443.833575] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2445.219774] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2445.309156] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2447.841228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2447.844391] Lustre: Skipped 2 previous similar messages [ 2452.106405] Lustre: server umount lustre-OST0000 complete [ 2452.964984] LustreError: 153382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.969949] LustreError: 153382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2458.082296] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2458.085561] LustreError: Skipped 1 previous similar message [ 2459.422590] Lustre: server umount lustre-OST0001 complete [ 2466.908826] Lustre: server umount lustre-MDT0000 complete [ 2468.242100] LustreError: 153371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663007 with bad export cookie 7504650629074085909 [ 2468.245196] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2468.246165] LustreError: 153371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2468.355605] Lustre: server umount lustre-MDT0001 complete [ 2476.020444] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 22:23:34 (1781663014) [ 2476.509410] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2476.985926] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2477.607124] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 22:23:36 (1781663016) [ 2479.693939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.785629] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2479.795772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2479.863385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2479.866816] Lustre: Skipped 6 previous similar messages [ 2481.080219] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2484.047431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2484.077592] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2484.089642] Lustre: 156926:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2485.683648] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2487.972228] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2489.672052] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2492.002273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2492.072497] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2494.059331] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2496.409885] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2497.886270] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2499.591595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2500.065643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2501.617250] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2501.693440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2504.674489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2504.676750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.679605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2504.682140] Lustre: Skipped 5 previous similar messages [ 2504.683814] Lustre: Skipped 11 previous similar messages [ 2508.445222] Lustre: server umount lustre-OST0000 complete [ 2522.115746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2524.202888] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2525.350445] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.357603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.357652] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2525.364436] Lustre: Skipped 1 previous similar message [ 2526.651171] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2528.179506] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2528.256673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2529.634795] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2529.708813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2531.509764] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88ce43385800.ost_server_uuid 50 [ 2531.982723] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88ce43385800.ost_server_uuid in FULL state after 0 sec [ 2532.042540] LustreError: 156930:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.43@tcp) returned error from blocking AST (req@000000000de3b62d x1868208604899072 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000017ebe07b/0x6825dffc5d5f7161 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.43@tcp remote: 0xf31bafa910f34abc expref: 5 pid: 156960 timeout: 2632 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2532.053584] LustreError: lustre-MDT0000: A client on nid 192.168.203.43@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2532.056968] LustreError: 156921:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.43@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000017ebe07b/0x6825dffc5d5f7161 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.43@tcp remote: 0xf31bafa910f34abc expref: 6 pid: 156960 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2535.907323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2535.914556] LustreError: Skipped 2 previous similar messages [ 2539.238210] Lustre: server umount lustre-OST0000 complete [ 2546.659455] Lustre: server umount lustre-MDT0000 complete [ 2547.903340] LustreError: 160207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663087 with bad export cookie 7504650629074087092 [ 2547.907145] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2547.910451] LustreError: 160207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2554.129509] Lustre: server umount lustre-MDT0001 complete [ 2557.637856] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2558.728884] Key type lgssc unregistered [ 2558.849361] LNet: 163040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2558.852539] LNetError: 163040:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2558.860267] LNet: Removed LNI 192.168.203.143@tcp [ 2559.236702] Key type .llcrypt unregistered [ 2559.237716] Key type ._llcrypt unregistered [ 2569.528752] Key type ._llcrypt registered [ 2569.530077] Key type .llcrypt registered [ 2569.569333] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2570.002059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2570.009722] alg: No test for adler32 (adler32-zlib) [ 2570.872615] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 2570.966957] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2572.551102] Key type lgssc registered [ 2573.115267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2576.790751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2576.802332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2577.996580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.879399] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2583.660765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.799171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2585.152783] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2588.135806] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2590.583098] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2593.286089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2593.420724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2595.407163] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2597.988172] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2603.691496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2610.554792] Lustre: Failing over lustre-MDT0000 [ 2610.685246] Lustre: server umount lustre-MDT0000 complete [ 2614.241046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.242650] LustreError: 164426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2614.247106] Lustre: Skipped 2 previous similar messages [ 2614.252763] LustreError: 164426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2619.360804] LustreError: 165158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.370648] LustreError: 165158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2621.576780] LustreError: 164411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.43@tcp arrived at 1781663160 with bad export cookie 11107181456168520171 [ 2621.581239] LustreError: 164411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2624.069152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.104437] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2624.191381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.217433] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2625.500282] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2626.454862] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2626.458801] Lustre: lustre-MDT0000: Denying connection for new client 415a849b-0150-41b7-a90c-59bf71e0e6fa (at 192.168.203.43@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2629.616248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2634.723834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2634.727322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2634.732945] Lustre: Skipped 1 previous similar message [ 2639.402677] Lustre: server umount lustre-OST0000 complete [ 2640.600778] LustreError: 168177:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2640.604075] Lustre: 167528:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2640.606426] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2640.612123] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2640.617243] LustreError: 167528:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000443af925 x1868209177267584/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2640.625422] LustreError: 167528:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2640.629780] LustreError: 167528:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2640.634580] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2640.735469] Lustre: server umount lustre-MDT0000 complete [ 2641.919157] LustreError: 165166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663181 with bad export cookie 11107181456168520528 [ 2641.923717] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2641.924240] LustreError: 163740:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c5c45f5c x1868209177268480/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2648.123113] Lustre: server umount lustre-MDT0001 complete [ 2651.742527] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2652.783542] Key type lgssc unregistered [ 2652.921348] LNet: 168928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2652.924666] LNetError: 168928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2652.936401] LNet: Removed LNI 192.168.203.143@tcp [ 2653.279869] Key type .llcrypt unregistered [ 2653.281417] Key type ._llcrypt unregistered [ 2662.579888] Key type ._llcrypt registered [ 2662.580896] Key type .llcrypt registered [ 2662.617803] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2662.999708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2663.019524] alg: No test for adler32 (adler32-zlib) [ 2663.886951] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 2663.969888] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2665.543129] Key type lgssc registered [ 2666.039903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2668.682498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2668.689152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.870637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2671.365756] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2674.475493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.591534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2676.283323] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2679.209172] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2681.420289] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2684.137392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2684.254231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2686.112649] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2688.422533] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2696.555203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2701.791783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2701.793943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2701.797784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2704.867232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2704.869395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2704.879118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2707.545386] Lustre: server umount lustre-OST0000 complete [ 2709.984285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.984907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2709.987848] Lustre: Skipped 1 previous similar message [ 2709.989491] Lustre: Skipped 1 previous similar message [ 2715.104131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2715.106081] Lustre: Skipped 2 previous similar messages [ 2720.225177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2720.229458] Lustre: Skipped 1 previous similar message [ 2723.295112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2723.366806] Lustre: server umount lustre-MDT0000 complete [ 2724.731516] LustreError: 170299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663264 with bad export cookie 11842312211717906315 [ 2724.736254] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2724.739712] LustreError: 170299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2724.862832] Lustre: server umount lustre-MDT0001 complete [ 2728.504009] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2729.559053] Key type lgssc unregistered [ 2729.699566] LNet: 173870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2729.707499] LNetError: 173870:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2729.721405] LNet: Removed LNI 192.168.203.143@tcp [ 2730.002785] Key type .llcrypt unregistered [ 2730.003859] Key type ._llcrypt unregistered [ 2742.029780] Key type ._llcrypt registered [ 2742.030927] Key type .llcrypt registered [ 2742.072785] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2742.393199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2742.408705] alg: No test for adler32 (adler32-zlib) [ 2743.274272] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 2743.358970] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2744.943212] Key type lgssc registered [ 2745.428672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2748.718436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2748.728907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.894938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2751.525734] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2754.833551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.982251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2756.656641] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2759.846312] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2762.287962] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2765.231559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2765.397115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.940106] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2771.366816] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2775.653789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2785.251872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2785.253447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2785.258437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2785.267029] Lustre: Skipped 1 previous similar message [ 2786.508649] Lustre: server umount lustre-OST0000 complete [ 2790.370378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2790.371852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2790.378490] Lustre: Skipped 1 previous similar message [ 2790.385288] Lustre: Skipped 2 previous similar messages [ 2794.289821] Lustre: server umount lustre-MDT0000 complete [ 2795.488813] LustreError: 176286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2795.499318] LustreError: 176286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2796.056976] LustreError: 177390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663335 with bad export cookie 14028150507038498249 [ 2796.063142] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2796.069388] LustreError: 177390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2796.206979] Lustre: server umount lustre-MDT0001 complete [ 2800.031285] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2801.221512] Key type lgssc unregistered [ 2801.372666] LNet: 179099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2801.378724] LNetError: 179099:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2801.390585] LNet: Removed LNI 192.168.203.143@tcp [ 2801.868276] Key type .llcrypt unregistered [ 2801.870563] Key type ._llcrypt unregistered [ 2811.194936] Key type ._llcrypt registered [ 2811.195894] Key type .llcrypt registered [ 2811.232300] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2811.539400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2811.591649] alg: No test for adler32 (adler32-zlib) [ 2812.439418] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 2812.513800] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2814.087129] Key type lgssc registered [ 2814.431715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2817.714285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2817.720310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.854076] Lustre: *** cfs_fail_loc=135, val=0*** [ 2818.894384] LustreError: 180599:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2818.897881] LustreError: 180599:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.143@tcp: cfg command failed: rc = -2 [ 2818.900946] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2818.900946] [ 2818.904423] LustreError: MGC192.168.203.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2818.912179] LustreError: 180560:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2818.916080] LustreError: 180560:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2818.919479] LustreError: 180560:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2818.922398] LustreError: 180560:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2818.958721] Lustre: server umount lustre-MDT0000 complete [ 2818.960697] LustreError: 180560:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2821.180781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2821.344512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2822.610116] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2825.357701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2825.471639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2826.800646] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2829.110595] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2830.822997] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2831.455844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.456476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2831.461954] Lustre: Skipped 1 previous similar message [ 2835.552283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2835.554360] Lustre: Skipped 2 previous similar messages [ 2837.592902] Lustre: server umount lustre-MDT0000 complete [ 2838.759146] LustreError: 180954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663378 with bad export cookie 1945513169673362350 [ 2838.762044] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2838.762624] LustreError: 180954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2838.883709] Lustre: server umount lustre-MDT0001 complete [ 2842.448675] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2843.371665] Key type lgssc unregistered [ 2843.498391] LNet: 183288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2843.501266] LNetError: 183288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2843.508291] LNet: Removed LNI 192.168.203.143@tcp [ 2843.785696] Key type .llcrypt unregistered [ 2843.786905] Key type ._llcrypt unregistered [ 2850.560818] Key type ._llcrypt registered [ 2850.561839] Key type .llcrypt registered [ 2850.599684] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2850.692206] Key type .llcrypt unregistered [ 2850.693523] Key type ._llcrypt unregistered [ 2857.154934] Key type ._llcrypt registered [ 2857.156020] Key type .llcrypt registered [ 2857.191446] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2857.489701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2857.527963] alg: No test for adler32 (adler32-zlib) [ 2858.390779] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 2858.476483] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2860.055195] Key type lgssc registered [ 2860.405361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2862.843544] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2862.849780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2878.303403] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2883.423145] LustreError: 185394:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 2910.047166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2911.962990] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2914.293963] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2916.539356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2916.720624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2916.741935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2918.082495] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2920.894344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2921.028357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2922.372949] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2924.755580] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2926.528717] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2927.500941] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2940.898439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.900051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2940.902341] Lustre: Skipped 1 previous similar message [ 2943.563315] Lustre: server umount lustre-OST0000 complete [ 2946.016808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.021042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2946.022978] Lustre: Skipped 2 previous similar messages [ 2951.136280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2951.139394] Lustre: Skipped 1 previous similar message [ 2956.256107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2956.258164] Lustre: Skipped 1 previous similar message [ 2959.327152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2959.413500] Lustre: server umount lustre-MDT0000 complete [ 2960.666422] LustreError: 185402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663499 with bad export cookie 17593613758146353605 [ 2960.670699] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2960.672331] LustreError: 185402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2960.788409] Lustre: server umount lustre-MDT0001 complete [ 2964.695921] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2965.767558] Key type lgssc unregistered [ 2965.886300] LNet: 189502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2965.889502] LNetError: 189502:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2965.899297] LNet: Removed LNI 192.168.203.143@tcp [ 2966.225634] Key type .llcrypt unregistered [ 2966.226791] Key type ._llcrypt unregistered [ 2976.269781] Key type ._llcrypt registered [ 2976.271315] Key type .llcrypt registered [ 2976.336562] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2976.664130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2976.670345] alg: No test for adler32 (adler32-zlib) [ 2977.526317] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 2977.615066] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2979.199163] Key type lgssc registered [ 2979.831572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2982.508010] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2982.516561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.686831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2985.001884] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2989.050532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2989.221543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2990.803423] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2993.580331] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2995.699157] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2998.225421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2998.363428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3000.412100] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3002.993966] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3008.617642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3014.548704] Lustre: Failing over lustre-MDT0000 [ 3014.650874] Lustre: server umount lustre-MDT0000 complete [ 3015.136731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3015.140818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3015.148397] LustreError: 190892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3015.160961] LustreError: 190892:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3018.450038] LustreError: 193389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.234331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3019.240435] Lustre: Skipped 1 previous similar message [ 3023.555555] LustreError: 193389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3023.569461] LustreError: 193389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3028.128442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3028.181210] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3028.298120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3028.343416] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3028.675847] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3029.864001] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3031.498516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3033.580085] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3033.592403] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3033.611207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3036.448604] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3045.855721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3045.857889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.861580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3048.933549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3048.933672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3048.937035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3048.938127] LustreError: Skipped 1 previous similar message [ 3048.942593] Lustre: Skipped 1 previous similar message [ 3050.213673] Lustre: server umount lustre-OST0000 complete [ 3054.050492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3054.050771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3054.066370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3054.070410] Lustre: Skipped 1 previous similar message [ 3058.008733] Lustre: server umount lustre-MDT0000 complete [ 3059.168827] LustreError: 190893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.177637] LustreError: 190893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3059.552460] LustreError: 190872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663598 with bad export cookie 13015175164527614853 [ 3059.555362] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3059.558165] LustreError: 190872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3059.705460] Lustre: server umount lustre-MDT0001 complete [ 3063.948229] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 3065.304855] Key type lgssc unregistered [ 3065.462352] LNet: 195791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3065.464804] LNetError: 195791:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3065.472768] LNet: Removed LNI 192.168.203.143@tcp [ 3065.953960] Key type .llcrypt unregistered [ 3065.956266] Key type ._llcrypt unregistered [ 3075.566769] Key type ._llcrypt registered [ 3075.568476] Key type .llcrypt registered [ 3075.615327] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3075.920722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3075.957063] alg: No test for adler32 (adler32-zlib) [ 3076.816075] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3076.903974] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 3078.511128] Key type lgssc registered [ 3078.984713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3082.415375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3082.426800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3083.592750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.075748] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3088.672855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.800158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3090.116988] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3092.663263] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3094.667821] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3097.468731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3097.604499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.427164] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3102.382897] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3107.878086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3108.517903] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3112.037056] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3124.870888] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3133.411274] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3133.413088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.414811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3133.421237] Lustre: Skipped 1 previous similar message [ 3137.109808] Lustre: server umount lustre-OST0000 complete [ 3138.528843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3138.529634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3138.533725] Lustre: Skipped 1 previous similar message [ 3138.536023] Lustre: Skipped 1 previous similar message [ 3143.648791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3143.650986] Lustre: Skipped 1 previous similar message [ 3144.552107] Lustre: server umount lustre-MDT0000 complete [ 3145.944428] LustreError: 197161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663685 with bad export cookie 3286954806982142677 [ 3145.949785] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3145.950047] LustreError: 197161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3146.096172] Lustre: server umount lustre-MDT0001 complete [ 3149.526923] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 3150.542347] Key type lgssc unregistered [ 3150.678339] LNet: 201042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3150.681558] LNetError: 201042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3150.689328] LNet: Removed LNI 192.168.203.143@tcp [ 3150.992637] Key type .llcrypt unregistered [ 3150.993747] Key type ._llcrypt unregistered [ 3158.216629] Key type ._llcrypt registered [ 3158.217628] Key type .llcrypt registered [ 3158.254136] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3158.601753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3158.647467] alg: No test for adler32 (adler32-zlib) [ 3159.507619] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3159.588433] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 3161.167141] Key type lgssc registered [ 3161.523758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3163.995735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3164.001278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3165.155207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3166.422810] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3169.461717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.587621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3170.815366] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3173.073908] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3174.780994] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3176.958967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3177.128442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3179.152693] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3181.565185] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3191.469688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3193.991721] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3206.459906] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3212.082482] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3219.992128] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3230.687872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3230.689950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.693547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3232.737679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3232.739475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3232.741036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3233.932498] Lustre: server umount lustre-OST0000 complete [ 3236.319698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3236.322054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3236.325842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3236.327396] Lustre: Skipped 1 previous similar message [ 3241.305428] Lustre: server umount lustre-MDT0000 complete [ 3242.481176] LustreError: 202278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663781 with bad export cookie 1269448402060658184 [ 3242.484777] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3242.485395] LustreError: 202278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3242.589599] Lustre: server umount lustre-MDT0001 complete [ 3245.967910] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 3246.927230] Key type lgssc unregistered [ 3247.072329] LNet: 207771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3247.075208] LNetError: 207771:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3247.085284] LNet: Removed LNI 192.168.203.143@tcp [ 3247.367064] Key type .llcrypt unregistered [ 3247.368280] Key type ._llcrypt unregistered [ 3256.007501] Key type ._llcrypt registered [ 3256.008510] Key type .llcrypt registered [ 3256.047848] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3256.361475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3256.367515] alg: No test for adler32 (adler32-zlib) [ 3257.219219] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3257.302514] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 3258.879125] Key type lgssc registered [ 3259.232493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3261.837468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3261.846085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.063735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.401818] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3267.180425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3267.296922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3268.630903] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3270.952496] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3272.671391] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3274.877093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3275.003744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3276.906214] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3279.279909] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3288.357461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3293.664637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3293.665964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3293.673576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3293.675610] Lustre: Skipped 1 previous similar message [ 3295.054630] Lustre: server umount lustre-OST0000 complete [ 3298.271749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3298.276116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.281698] Lustre: Skipped 1 previous similar message [ 3298.285406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3298.390698] Lustre: server umount lustre-MDT0000 complete [ 3298.784745] LustreError: 209155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.789723] LustreError: 209155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3299.566378] LustreError: 209140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663838 with bad export cookie 2974334257842884153 [ 3299.568967] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3299.573133] LustreError: 209140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3299.697995] Lustre: server umount lustre-MDT0001 complete [ 3303.145779] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 3304.114281] Key type lgssc unregistered [ 3304.237340] LNet: 212860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3304.240187] LNetError: 212860:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3304.249343] LNet: Removed LNI 192.168.203.143@tcp [ 3304.528684] Key type .llcrypt unregistered [ 3304.530313] Key type ._llcrypt unregistered [ 3311.677201] Key type ._llcrypt registered [ 3311.678189] Key type .llcrypt registered [ 3311.720784] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3312.124783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3312.138232] alg: No test for adler32 (adler32-zlib) [ 3312.994978] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3313.077050] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 3314.655169] Key type lgssc registered [ 3315.048108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3317.675607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3317.682111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3318.848147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3320.153249] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3323.020394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.152845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3324.507333] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3326.903976] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3328.634609] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3330.769812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3330.907516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3332.798360] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3335.140509] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3344.230063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3353.567844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3353.570467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.575328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3354.593239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.593320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3354.593542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3354.605392] Lustre: Skipped 1 previous similar message [ 3356.043986] Lustre: server umount lustre-OST0000 complete [ 3359.199868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3359.202512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3359.206930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3363.488304] Lustre: server umount lustre-MDT0000 complete [ 3364.797482] LustreError: 214082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663904 with bad export cookie 13071224569059294004 [ 3364.800846] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3364.805462] LustreError: 214082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3364.943214] Lustre: server umount lustre-MDT0001 complete [ 3368.694032] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 3369.663515] Key type lgssc unregistered [ 3369.783346] LNet: 217805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3369.787589] LNetError: 217805:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3369.798290] LNet: Removed LNI 192.168.203.143@tcp [ 3370.083512] Key type .llcrypt unregistered [ 3370.085461] Key type ._llcrypt unregistered [ 3377.056437] Key type ._llcrypt registered [ 3377.057894] Key type .llcrypt registered [ 3377.093750] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3377.422679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3377.459065] alg: No test for adler32 (adler32-zlib) [ 3378.311080] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3378.399786] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 3379.991106] Key type lgssc registered [ 3380.336152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3382.825869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3382.833317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3383.982704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3385.273316] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3388.000180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3388.128941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3389.413540] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3391.670981] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3393.405305] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3395.511068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3395.660729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3397.513178] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3399.839715] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3408.996200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3412.486450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3412.515759] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3412.591442] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3414.454634] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3416.799836] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3417.368362] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3427.721679] Lustre: Permanently deactivating lustre-OST0001 [ 3427.727319] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3428.853310] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3439.589525] Lustre: Permanently reactivating lustre-OST0001 [ 3439.595064] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3439.598296] Lustre: Skipped 2 previous similar messages [ 3447.797062] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3447.801370] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3447.804274] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3447.808126] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3447.818520] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3451.874586] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3451.877439] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3451.878197] Lustre: Skipped 2 previous similar messages [ 3455.877301] Lustre: server umount lustre-OST0001 complete [ 3456.997314] LustreError: 220866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3457.001932] LustreError: 220866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3459.551773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3459.555581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.561847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3459.564569] Lustre: Skipped 1 previous similar message [ 3460.576106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3460.579989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3462.115513] LustreError: 220866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3462.120654] LustreError: 220866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3467.232863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3467.235028] Lustre: Skipped 3 previous similar messages [ 3467.235816] LustreError: 220866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3467.240408] LustreError: 220866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3471.327100] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3471.375823] Lustre: server umount lustre-OST0000 complete [ 3475.423818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3475.427630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3475.435228] Lustre: Skipped 1 previous similar message [ 3475.437498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3478.680722] Lustre: server umount lustre-MDT0000 complete [ 3479.891255] LustreError: 219029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664019 with bad export cookie 16239210979072207607 [ 3479.894417] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3479.895957] LustreError: 219029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3480.016319] Lustre: server umount lustre-MDT0001 complete [ 3483.466112] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 3484.438504] Key type lgssc unregistered [ 3484.554327] LNet: 224111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3484.556546] LNetError: 224111:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3484.564411] LNet: Removed LNI 192.168.203.143@tcp [ 3484.839252] Key type .llcrypt unregistered [ 3484.840332] Key type ._llcrypt unregistered [ 3498.374770] Key type ._llcrypt registered [ 3498.375827] Key type .llcrypt registered [ 3498.412209] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3498.736973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3498.789968] alg: No test for adler32 (adler32-zlib) [ 3499.655877] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3499.746923] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 3501.327129] Key type lgssc registered [ 3501.712198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3504.237321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3504.245404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.338139] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3505.345544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3505.405169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3506.769176] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3509.462712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3509.492647] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3509.502796] Lustre: 226292:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3509.600414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3510.871873] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3513.191336] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3514.934822] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3517.113254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3517.181495] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3517.247975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3518.735924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3519.145910] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3521.487442] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3523.038462] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3531.594768] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3537.123538] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3546.849749] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3562.078180] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3562.081604] Lustre: Skipped 1 previous similar message [ 3582.678556] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3582.680807] Lustre: Skipped 2 previous similar messages [ 3620.262386] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3620.265457] Lustre: Skipped 4 previous similar messages [ 3687.906301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3687.906430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3687.907575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3687.908956] LustreError: Skipped 1 previous similar message [ 3687.913307] Lustre: Skipped 1 previous similar message [ 3690.510871] Lustre: server umount lustre-OST0000 complete [ 3693.025458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3693.026434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3693.031232] Lustre: Skipped 1 previous similar message [ 3693.037754] Lustre: Skipped 2 previous similar messages [ 3698.144129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3698.146429] Lustre: Skipped 1 previous similar message [ 3703.264234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3703.266144] Lustre: Skipped 1 previous similar message [ 3706.335120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3706.402322] Lustre: server umount lustre-MDT0000 complete [ 3707.567130] LustreError: 229029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664246 with bad export cookie 4620441324717823075 [ 3707.570581] LustreError: 229029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3707.571179] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3707.684500] Lustre: server umount lustre-MDT0001 complete [ 3711.002456] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 3712.014055] Key type lgssc unregistered [ 3712.131299] LNet: 231082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3712.133686] LNetError: 231082:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3712.139258] LNet: Removed LNI 192.168.203.143@tcp [ 3712.425309] Key type .llcrypt unregistered [ 3712.426628] Key type ._llcrypt unregistered [ 3720.969960] Key type ._llcrypt registered [ 3720.971524] Key type .llcrypt registered [ 3721.013203] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3721.417448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3721.432749] alg: No test for adler32 (adler32-zlib) [ 3722.282280] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3722.365267] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 3723.943112] Key type lgssc registered [ 3724.275920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3726.802654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3726.812615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3727.982573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3729.248152] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3732.306390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.474960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3733.876416] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3736.173323] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3737.905439] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3740.026036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3740.164223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3742.099370] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3744.598320] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3753.512891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 3762.656315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3762.660857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.669204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3763.681791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3763.681917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3763.682420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.684327] Lustre: Skipped 1 previous similar message [ 3764.875189] Lustre: server umount lustre-OST0000 complete [ 3768.288108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3768.292518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3768.299360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3772.304643] Lustre: server umount lustre-MDT0000 complete [ 3773.599196] LustreError: 235070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664312 with bad export cookie 11431328802060135702 [ 3773.601316] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3773.605205] LustreError: 235070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3773.738749] Lustre: server umount lustre-MDT0001 complete [ 3777.501477] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 3778.689838] Key type lgssc unregistered [ 3778.827624] LNet: 236177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3778.830603] LNetError: 236177:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3778.839451] LNet: Removed LNI 192.168.203.143@tcp [ 3779.174171] Key type .llcrypt unregistered [ 3779.176190] Key type ._llcrypt unregistered [ 3787.129244] Key type ._llcrypt registered [ 3787.130231] Key type .llcrypt registered [ 3787.163802] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3787.478601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3787.505739] alg: No test for adler32 (adler32-zlib) [ 3788.358760] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3788.437058] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 3790.015101] Key type lgssc registered [ 3790.354963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4056.807428] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 4057.782279] Key type lgssc unregistered [ 4057.907351] LNet: 238174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4057.909766] LNetError: 238174:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4057.916294] LNet: Removed LNI 192.168.203.143@tcp [ 4058.199289] Key type .llcrypt unregistered [ 4058.200251] Key type ._llcrypt unregistered [ 4066.499390] Key type ._llcrypt registered [ 4066.500455] Key type .llcrypt registered [ 4066.538646] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 4074.329384] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 4074.684204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4074.715811] alg: No test for adler32 (adler32-zlib) [ 4075.571279] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 4075.652383] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 4077.231203] Key type lgssc registered [ 4077.699055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4087.515538] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3951 sec ========== 22:50:26 (1781664626) [ 4088.030509] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:50:26 (1781664626) === [ 4089.000289] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:50:27 (1781664627) === [ 4100.820665] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 4101.702904] Key type lgssc unregistered [ 4101.819258] LNet: 243379:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4101.821506] LNetError: 243379:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4101.830246] LNet: Removed LNI 192.168.203.143@tcp [ 4102.104429] Key type .llcrypt unregistered [ 4102.105738] Key type ._llcrypt unregistered