[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1765425062 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002426] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007354] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010205] LSM: Security Framework initializing [ 0.012043] Yama: becoming mindful. [ 0.013026] SELinux: Initializing. [ 0.014057] *** VALIDATE selinux *** [ 0.023172] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028306] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031108] *** VALIDATE tmpfs *** [ 0.033314] *** VALIDATE proc *** [ 0.034229] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.036305] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038040] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040024] Spectre V2 : User space: Vulnerable [ 0.041007] Speculative Store Bypass: Vulnerable [ 0.044131] debug: unmapping init [mem 0xffffffffb3459000-0xffffffffb3460fff] [ 0.046235] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047800] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048019] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051015] ... value mask: 0000ffffffffffff [ 0.052010] ... max period: 00007fffffffffff [ 0.053011] ... fixed-purpose events: 3 [ 0.054008] ... event mask: 000000070000000f [ 0.055330] rcu: Hierarchical SRCU implementation. [ 0.057614] smp: Bringing up secondary CPUs ... [ 0.058621] x86: Booting SMP configuration: [ 0.059019] .... node #0, CPUs: #1 #2 #3 [ 0.068181] smp: Brought up 1 node, 4 CPUs [ 0.070014] smpboot: Max logical packages: 1 [ 0.071012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.110176] node 0 deferred pages initialised in 37ms [ 0.113312] devtmpfs: initialized [ 0.114248] x86/mm: Memory block size: 128MB [ 0.117067] gcov: version magic: 0x41383552 [ 0.119261] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.120114] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.121264] pinctrl core: initialized pinctrl subsystem [ 0.122186] [ 0.122789] ************************************************************* [ 0.123014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.124029] ** ** [ 0.125008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.126009] ** ** [ 0.127009] ** This means that this kernel is built to expose internal ** [ 0.128010] ** IOMMU data structures, which may compromise security on ** [ 0.129009] ** your system. ** [ 0.130012] ** ** [ 0.131012] ** If you see this message and you are not debugging the ** [ 0.132011] ** kernel, report this immediately to your vendor! ** [ 0.133011] ** ** [ 0.134011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135010] ************************************************************* [ 0.136772] NET: Registered protocol family 16 [ 0.137424] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.138110] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.139061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.140854] cpuidle: using governor menu [ 0.142512] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.144699] PCI: Using configuration type 1 for base access [ 0.147141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.152357] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.153000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.154124] cryptd: max_cpu_qlen set to 1000 [ 0.156020] ACPI: Added _OSI(Module Device) [ 0.157013] ACPI: Added _OSI(Processor Device) [ 0.158008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.159008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.163874] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.167306] ACPI: Interpreter enabled [ 0.168068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.169012] ACPI: Using IOAPIC for interrupt routing [ 0.170206] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.171563] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.182208] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.183058] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.184032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.185116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.187711] acpiphp: Slot [2] registered [ 0.188126] acpiphp: Slot [5] registered [ 0.189118] acpiphp: Slot [6] registered [ 0.190197] acpiphp: Slot [7] registered [ 0.191081] acpiphp: Slot [8] registered [ 0.192166] acpiphp: Slot [9] registered [ 0.193090] acpiphp: Slot [10] registered [ 0.194085] acpiphp: Slot [3] registered [ 0.195065] acpiphp: Slot [4] registered [ 0.196076] acpiphp: Slot [11] registered [ 0.197094] acpiphp: Slot [12] registered [ 0.198080] acpiphp: Slot [13] registered [ 0.199128] acpiphp: Slot [14] registered [ 0.200076] acpiphp: Slot [15] registered [ 0.201105] acpiphp: Slot [16] registered [ 0.202161] acpiphp: Slot [17] registered [ 0.203100] acpiphp: Slot [18] registered [ 0.204071] acpiphp: Slot [19] registered [ 0.205071] acpiphp: Slot [20] registered [ 0.206067] acpiphp: Slot [21] registered [ 0.207072] acpiphp: Slot [22] registered [ 0.208071] acpiphp: Slot [23] registered [ 0.209078] acpiphp: Slot [24] registered [ 0.210093] acpiphp: Slot [25] registered [ 0.211081] acpiphp: Slot [26] registered [ 0.212081] acpiphp: Slot [27] registered [ 0.213099] acpiphp: Slot [28] registered [ 0.214159] acpiphp: Slot [29] registered [ 0.215082] acpiphp: Slot [30] registered [ 0.216094] acpiphp: Slot [31] registered [ 0.217056] PCI host bridge to bus 0000:00 [ 0.218016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.219018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.220020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.221020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.222016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.223024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.224185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.225744] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.229332] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.239018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.244026] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.246019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.248114] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.250013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.253709] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.257892] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.260078] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.263026] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.270016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.284054] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.290019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.297715] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.317021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.336042] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.350000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.385171] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.408018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.422015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.460016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.472350] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.485017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.494061] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.519019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.531000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.534015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.540014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.554015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.561796] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.568016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.573015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.589017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.598152] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.605015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.612017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.632029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.651609] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.661101] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.668079] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.672388] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.677227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.686090] iommu: Default domain type: Passthrough [ 0.687561] SCSI subsystem initialized [ 0.689194] ACPI: bus type USB registered [ 0.691110] usbcore: registered new interface driver usbfs [ 0.693098] usbcore: registered new interface driver hub [ 0.695108] usbcore: registered new device driver usb [ 0.697163] pps_core: LinuxPPS API ver. 1 registered [ 0.699009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.702071] PTP clock support registered [ 0.704094] EDAC MC: Ver: 3.0.0 [ 0.706081] PCI: Using ACPI for IRQ routing [ 0.706892] NetLabel: Initializing [ 0.707011] NetLabel: domain hash size = 128 [ 0.708008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.710078] NetLabel: unlabeled traffic allowed by default [ 0.712282] vgaarb: loaded [ 0.713464] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.715013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.722345] clocksource: Switched to clocksource kvm-clock [ 0.847811] VFS: Disk quotas dquot_6.6.0 [ 0.849542] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.852127] *** VALIDATE ramfs *** [ 0.853757] *** VALIDATE hugetlbfs *** [ 0.855848] pnp: PnP ACPI init [ 0.858708] pnp: PnP ACPI: found 6 devices [ 0.876246] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.880043] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.882628] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.884368] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.886819] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.888901] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.891759] NET: Registered protocol family 2 [ 0.894820] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.900105] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.904295] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.909748] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.914326] TCP: Hash tables configured (established 65536 bind 65536) [ 0.917074] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.920489] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.923902] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.927399] NET: Registered protocol family 1 [ 0.930974] RPC: Registered named UNIX socket transport module. [ 0.932825] RPC: Registered udp transport module. [ 0.934247] RPC: Registered tcp transport module. [ 0.935729] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.937822] NET: Registered protocol family 44 [ 0.938960] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.941251] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.943591] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.946071] PCI: CLS 0 bytes, default 64 [ 0.948152] Unpacking initramfs... [ 2.750176] debug: unmapping init [mem 0xffff946fbcc54000-0xffff946fbffbffff] [ 2.764435] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.766677] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.769612] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.571373] Initialise system trusted keyrings [ 3.572993] Key type blacklist registered [ 3.574805] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.584227] zbud: loaded [ 3.587124] *** VALIDATE nfs *** [ 3.588292] *** VALIDATE nfs4 *** [ 3.590104] pstore: using deflate compression [ 3.594183] Platform Keyring initialized [ 3.726204] NET: Registered protocol family 38 [ 3.728203] Key type asymmetric registered [ 3.730300] Asymmetric key parser 'x509' registered [ 3.733891] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.740544] io scheduler mq-deadline registered [ 3.742817] io scheduler kyber registered [ 3.746128] io scheduler bfq registered [ 3.749152] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.755548] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.759130] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.763270] ACPI: Power Button [PWRF] [ 3.770788] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.781231] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.808540] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.824178] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.862688] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.940715] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.040135] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.116937] Non-volatile memory driver v1.3 [ 4.119135] Linux agpgart interface v0.103 [ 4.194637] virtio_blk virtio1: [vda] 145160 512-byte logical blocks (74.3 MB/70.9 MiB) [ 4.199249] vda: detected capacity change from 0 to 74321920 [ 4.451410] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.458604] vdb: detected capacity change from 0 to 1073741824 [ 4.560722] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.563698] vdc: detected capacity change from 0 to 2621440000 [ 4.623912] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.627431] vdd: detected capacity change from 0 to 2621440000 [ 4.643240] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.646044] vde: detected capacity change from 0 to 4294967296 [ 4.660778] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.664070] vdf: detected capacity change from 0 to 4294967296 [ 4.732239] libphy: Fixed MDIO Bus: probed [ 4.741716] usbcore: registered new interface driver usbserial_generic [ 4.744830] usbserial: USB Serial support registered for generic [ 4.748526] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.754135] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.756081] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.759621] mousedev: PS/2 mouse device common for all mice [ 4.763734] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.764960] rtc_cmos 00:05: RTC can wake from S4 [ 4.777898] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.781140] rtc_cmos 00:05: registered as rtc0 [ 4.783988] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.788417] intel_pstate: CPU model not supported [ 4.790821] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.795321] hid: raw HID events driver (C) Jiri Kosina [ 4.797444] usbcore: registered new interface driver usbhid [ 4.799477] usbhid: USB HID core driver [ 4.800989] drop_monitor: Initializing network drop monitor service [ 4.803468] Initializing XFRM netlink socket [ 4.805328] NET: Registered protocol family 10 [ 4.808142] Segment Routing with IPv6 [ 4.809895] NET: Registered protocol family 17 [ 4.812195] mpls_gso: MPLS GSO support [ 4.818980] RAS: Correctable Errors collector initialized. [ 4.821421] AVX version of gcm_enc/dec engaged. [ 4.822934] AES CTR mode by8 optimization enabled [ 4.919273] sched_clock: Marking stable (4919263170, 0)->(6735952250, -1816689080) [ 4.926476] registered taskstats version 1 [ 4.928812] Loading compiled-in X.509 certificates [ 4.931155] zswap: loaded using pool lzo/zbud [ 4.964157] Key type big_key registered [ 4.980560] Key type encrypted registered [ 4.982617] ima: No TPM chip found, activating TPM-bypass! [ 4.986133] ima: Allocated hash algorithm: sha1 [ 4.988491] ima: No architecture policies found [ 4.990242] evm: Initialising EVM extended attributes: [ 4.992301] evm: security.selinux [ 4.993708] evm: security.ima [ 4.994905] evm: security.capability [ 4.996507] evm: HMAC attrs: 0x1 [ 4.998979] rtc_cmos 00:05: setting system clock to 2026-07-03 09:31:47 UTC (1783071107) [ 5.006775] debug: unmapping init [mem 0xffffffffb4403000-0xffffffffb45fffff] [ 5.010270] debug: unmapping init [mem 0xffffffffb3182000-0xffffffffb3458fff] [ 5.018308] Write protecting the kernel read-only data: 28672k [ 5.022547] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 5.025250] debug: unmapping init [mem 0xffffffffb2114000-0xffffffffb21fffff] [ 5.144576] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.153714] systemd[1]: Detected virtualization kvm. [ 5.156177] systemd[1]: Detected architecture x86-64. [ 5.158313] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.188889] systemd[1]: No hostname configured. [ 5.192197] systemd[1]: Set hostname to . [ 5.194521] random: systemd: uninitialized urandom read (16 bytes read) [ 5.196885] systemd[1]: Initializing machine ID from random generator. [ 5.306885] random: ln: uninitialized urandom read (6 bytes read) [ 5.485441] random: systemd: uninitialized urandom read (16 bytes read) [ 5.487759] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.494868] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.503936] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started dracut cmdline hook. [ OK ] Started Create Static Device Nodes in /dev. Starting dracut pre-udev hook... [ OK ] Started Journal Service. [ 6.843867] device-mapper: uevent: version 1.0.3 [ 6.846363] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 8.379473] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.526165] virtio_net virtio0 ens2: renamed from eth0 [ 8.591749] scsi host0: ata_piix [ 9.003666] scsi host1: ata_piix [ 9.007890] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.012379] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.626363] random: crng init done [ 14.634711] random: 7 urandom warning(s) missed due to ratelimiting [ 14.701883] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 16.325826] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.783935] printk: systemd: 28 output lines suppressed due to ratelimiting [ 19.294787] SELinux: Disabled at runtime. [ 19.387923] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.395173] systemd[1]: Detected virtualization kvm. [ 19.396871] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.189724] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.193811] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.202280] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.207816] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.237390] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.259309] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.266724] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details.[ 21.483471] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ 22.581707] hrtimer: interrupt took 2976070 ns [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 24.416606] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.835244] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.887926] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.364230] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.477880] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 33.074037] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 34.002220] NFS: Registering the id_resolver key type [ 34.011350] Key type id_resolver registered [ 34.012954] Key type id_legacy registered [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg133-server login: [ 112.875287] libcfs: loading out-of-tree module taints kernel. [ 112.919444] Key type ._llcrypt registered [ 112.924442] Key type .llcrypt registered [ 113.058949] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 131.452941] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 133.043227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 133.071387] alg: No test for adler32 (adler32-zlib) [ 134.627710] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 135.461503] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 137.255509] Key type lgssc registered [ 139.203858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 152.820774] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 193.274513] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 205.549681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 205.574961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.843787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 206.897538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 206.970663] Lustre: lustre-MDT0000: new disk, initializing [ 207.048931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.073602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 211.139647] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.884316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 225.069413] Lustre: 6517:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 225.116513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 225.121709] Lustre: Skipped 1 previous similar message [ 225.221423] Lustre: lustre-MDT0001: new disk, initializing [ 225.329748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.383961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 225.394041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 229.998752] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.782459] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 244.407974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.616800] Lustre: lustre-OST0000: new disk, initializing [ 244.619745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 244.623884] Lustre: 8453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 244.677992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.899924] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.513988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 254.529805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 254.635195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 264.613190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 264.775786] Lustre: lustre-OST0001: new disk, initializing [ 264.780302] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 264.787876] Lustre: 9528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 264.866068] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 272.107859] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.454667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 273.462843] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 273.573787] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 284.717204] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.984754] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 303.023931] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing check_logdir /tmp/testlogs/ [ 308.578928] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing yml_node [ 313.727952] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 316.308659] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 318.901675] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 320.597609] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jul 3 05:37:01 EDT 2026 [ 337.733533] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 339.461912] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 355.811106] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.813846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 355.830383] Lustre: Skipped 3 previous similar messages [ 355.847793] Lustre: Skipped 3 previous similar messages [ 360.930884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.947322] Lustre: Skipped 3 previous similar messages [ 363.859938] Lustre: server umount lustre-MDT0000 complete [ 367.544115] LustreError: 8464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783071470 with bad export cookie 11195308829519826514 [ 367.552268] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 367.554102] LustreError: 8464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 371.169087] LustreError: 6523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.170470] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 371.170947] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.195578] LustreError: 6523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 371.246053] Lustre: Skipped 1 previous similar message [ 372.192990] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.213341] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 373.982610] Lustre: server umount lustre-MDT0001 complete [ 378.753365] Lustre: server umount lustre-OST0000 complete [ 382.673342] Lustre: server umount lustre-OST0001 complete [ 389.274712] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 399.202567] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 438.544844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.670492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 438.686503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 438.728059] Lustre: lustre-MDT0000: new disk, initializing [ 438.783096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.799562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 443.552730] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 455.396132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 455.556914] Lustre: 15080:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 455.670693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 455.679443] Lustre: Skipped 1 previous similar message [ 455.752791] Lustre: lustre-MDT0001: new disk, initializing [ 455.931245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 455.974255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 455.992176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 460.848944] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 470.312429] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 476.742440] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 484.123062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 484.437273] Lustre: lustre-OST0000: new disk, initializing [ 484.445772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 484.464539] Lustre: 17048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 484.607571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.585212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 485.592587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 485.809554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 491.684087] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 501.864613] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 507.408965] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 507.634804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 512.716726] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 512.988197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 516.576020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 516.583507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.585844] LustreError: Skipped 1 previous similar message [ 516.587673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 516.587680] Lustre: Skipped 3 previous similar messages [ 516.646445] Lustre: Skipped 1 previous similar message [ 521.353199] Lustre: server umount lustre-OST0000 complete [ 526.818165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.852859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 526.862766] Lustre: Skipped 2 previous similar messages [ 537.063594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 537.070490] Lustre: Skipped 3 previous similar messages [ 540.127482] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 540.247190] Lustre: server umount lustre-MDT0000 complete [ 542.182930] LustreError: 15091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.212546] LustreError: 15091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 543.967462] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783071646 with bad export cookie 11195308829519828047 [ 543.977272] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 543.997429] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 544.536710] Lustre: server umount lustre-MDT0001 complete [ 548.051618] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 05:40:48 (1783071648) [ 554.454667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.028175] LustreError: 19313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.158551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 560.350579] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 560.611331] LustreError: 19314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.489022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.344405] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 582.302498] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 587.690671] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 593.378833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 599.613361] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 606.318752] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 615.404871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.409538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 615.418046] Lustre: Skipped 1 previous similar message [ 615.418892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 615.436776] Lustre: Skipped 1 previous similar message [ 619.152950] Lustre: server umount lustre-OST0000 complete [ 625.632368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.645889] Lustre: Skipped 2 previous similar messages [ 628.871824] Lustre: server umount lustre-MDT0000 complete [ 630.757403] LustreError: 19310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.767308] LustreError: 19310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 632.523821] LustreError: 19294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783071735 with bad export cookie 11195308829519828929 [ 632.526340] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 632.537115] LustreError: 19294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 632.965605] Lustre: server umount lustre-MDT0001 complete [ 645.063843] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 648.359923] Key type lgssc unregistered [ 648.750621] LNet: 22848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.757217] LNetError: 22848:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 649.781050] LNet: Removed LNI 192.168.201.133@tcp [ 650.639282] Key type .llcrypt unregistered [ 650.640912] Key type ._llcrypt unregistered [ 674.175201] Key type ._llcrypt registered [ 674.179847] Key type .llcrypt registered [ 674.296178] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 675.699463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 675.737522] alg: No test for adler32 (adler32-zlib) [ 676.981614] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 677.231994] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 678.911263] Key type lgssc registered [ 680.430331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 691.917705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 691.940406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.642361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.855113] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 703.975043] LustreError: 24278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 703.999494] LustreError: 24278:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 709.102410] LustreError: 24277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.383653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.957412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 719.099592] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 728.439899] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 734.551509] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 742.371602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 742.749352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 750.670142] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 759.040160] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 764.372940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 778.721829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 778.741102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 778.757377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 779.744842] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 779.746242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 779.759082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.793071] Lustre: Skipped 1 previous similar message [ 783.322486] Lustre: server umount lustre-OST0000 complete [ 789.986700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 790.015673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 790.026357] Lustre: Skipped 1 previous similar message [ 793.917417] Lustre: server umount lustre-MDT0000 complete [ 795.104740] LustreError: 24278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 795.134092] LustreError: 24278:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 798.616328] LustreError: 24258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783071901 with bad export cookie 9771132292698034111 [ 798.625430] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 798.632052] LustreError: 24258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 798.990930] Lustre: server umount lustre-MDT0001 complete [ 810.871886] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 813.004408] Key type lgssc unregistered [ 813.235814] LNet: 28257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.245437] LNetError: 28257:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.314197] LNet: Removed LNI 192.168.201.133@tcp [ 815.617105] Key type .llcrypt unregistered [ 815.631020] Key type ._llcrypt unregistered [ 840.776094] Key type ._llcrypt registered [ 840.777643] Key type .llcrypt registered [ 840.876712] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 841.931360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 842.175148] alg: No test for adler32 (adler32-zlib) [ 843.238582] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 843.493611] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 845.192539] Key type lgssc registered [ 846.568392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 855.159637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 855.171524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.617982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.115578] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 866.790365] LustreError: 29673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 866.814178] LustreError: 29673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 869.959630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.421844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 874.209461] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 881.022212] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 888.215764] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 905.353788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 905.807518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.665327] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 922.279152] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 927.359995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 932.836180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 932.838575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.845899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 932.893996] Lustre: Skipped 1 previous similar message [ 934.426104] Lustre: server umount lustre-OST0000 complete [ 942.049994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 942.060990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.088544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 942.103160] Lustre: Skipped 1 previous similar message [ 943.077397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.098188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.104317] Lustre: Skipped 1 previous similar message [ 944.684885] Lustre: server umount lustre-MDT0000 complete [ 948.080031] LustreError: 29655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783072050 with bad export cookie 12154294221950165519 [ 948.094891] LustreError: 29655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 948.097001] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 948.441667] Lustre: server umount lustre-MDT0001 complete [ 959.727483] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 962.672749] Key type lgssc unregistered [ 963.007646] LNet: 33552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.016561] LNetError: 33552:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.042657] LNet: Removed LNI 192.168.201.133@tcp [ 964.180858] Key type .llcrypt unregistered [ 964.187980] Key type ._llcrypt unregistered [ 989.148799] Key type ._llcrypt registered [ 989.150285] Key type .llcrypt registered [ 989.278419] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 990.275849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 990.451488] alg: No test for adler32 (adler32-zlib) [ 991.513316] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 991.772717] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 993.455503] Key type lgssc registered [ 994.617842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1002.502544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1002.529812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.064128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1008.560498] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1014.251283] LustreError: 34967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1014.274735] LustreError: 34967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1016.790721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1017.110576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1021.446238] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1027.711782] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1032.241673] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1039.296209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1039.543488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1047.464195] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1048.789923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1057.713736] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1067.494334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1067.507421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1067.532216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1069.536327] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1069.539732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.547936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1069.550619] Lustre: Skipped 1 previous similar message [ 1071.632966] Lustre: server umount lustre-OST0000 complete [ 1078.752363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1078.755722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.775581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1082.678466] Lustre: server umount lustre-MDT0000 complete [ 1084.899307] LustreError: 36505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.915944] LustreError: 36505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1086.984544] LustreError: 34949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783072189 with bad export cookie 11330897646014527178 [ 1086.988516] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1086.992355] LustreError: 34949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1087.013929] LustreError: 34279:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting [ 1087.378886] Lustre: server umount lustre-MDT0001 complete [ 1097.960533] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1101.540448] Key type lgssc unregistered [ 1102.015647] LNet: 38499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1102.029747] LNetError: 38499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1102.058577] LNet: Removed LNI 192.168.201.133@tcp [ 1103.296297] Key type .llcrypt unregistered [ 1103.298961] Key type ._llcrypt unregistered [ 1127.218343] Key type ._llcrypt registered [ 1127.220409] Key type .llcrypt registered [ 1127.297868] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1128.647043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1128.687268] alg: No test for adler32 (adler32-zlib) [ 1129.688309] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 1129.916959] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1131.639210] Key type lgssc registered [ 1133.389762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1141.543718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1141.599909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.380078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1148.077849] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1153.511929] LustreError: 39911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1153.543262] LustreError: 39911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1158.630280] LustreError: 39912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.059639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.360228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1165.282256] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1172.878736] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1179.706954] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1186.284341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1186.680826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1188.751265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1194.934348] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1204.542199] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1212.662689] Lustre: lustre-OST0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 1214.432264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1214.435934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.446311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1215.462647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1215.472571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1215.486657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.530032] Lustre: server umount lustre-OST0000 complete [ 1241.059314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.090937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.100335] Lustre: Skipped 2 previous similar messages [ 1244.944443] Lustre: server umount lustre-MDT0000 complete [ 1246.176338] LustreError: 39911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.191090] LustreError: 39911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1248.955153] LustreError: 39893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783072351 with bad export cookie 4780703433245728010 [ 1248.969047] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.394069] Lustre: server umount lustre-MDT0001 complete [ 1266.829573] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1269.615984] Key type lgssc unregistered [ 1270.215255] LNet: 43621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1270.231059] LNetError: 43621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1270.247760] LNet: Removed LNI 192.168.201.133@tcp [ 1271.366195] Key type .llcrypt unregistered [ 1271.368386] Key type ._llcrypt unregistered [ 1300.415700] Key type ._llcrypt registered [ 1300.417788] Key type .llcrypt registered [ 1300.524658] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1301.716496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1301.887677] alg: No test for adler32 (adler32-zlib) [ 1302.944891] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 1303.279994] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1305.007176] Key type lgssc registered [ 1306.159773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1313.557115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1313.576565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.997883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1320.265914] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1324.515778] LustreError: 45038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.529414] LustreError: 45038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1328.934304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1329.335867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1334.394442] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1341.760623] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1347.884401] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1355.183927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1355.582035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.680682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1361.820609] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1368.062072] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1379.298812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1379.304667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1379.323710] Lustre: Skipped 2 previous similar messages [ 1379.344460] Lustre: Skipped 2 previous similar messages [ 1381.088468] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 1384.417178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1384.425814] Lustre: Skipped 1 previous similar message [ 1384.932463] Lustre: server umount lustre-MDT0000 complete [ 1386.193572] LustreError: 45034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.783799] LustreError: 45019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783072491 with bad export cookie 12263540541428179115 [ 1388.794732] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1388.804920] LustreError: 45019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1389.540529] LustreError: 47529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1389.544284] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.563076] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1389.567751] Lustre: Skipped 1 previous similar message [ 1393.576369] LustreError: 45034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.577647] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1393.601023] LustreError: 45034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1393.607962] Lustre: Skipped 1 previous similar message [ 1398.690462] LustreError: 45031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.716909] LustreError: 45031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1402.669882] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.33@tcp (stopping) [ 1402.677165] Lustre: Skipped 3 previous similar messages [ 1403.359276] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1403.823715] Lustre: server umount lustre-MDT0001 complete [ 1418.864881] Lustre: server umount lustre-OST0000 complete [ 1434.930314] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1437.706383] Key type lgssc unregistered [ 1438.090814] LNet: 48896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.098445] LNetError: 48896:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1438.115280] LNet: Removed LNI 192.168.201.133@tcp [ 1439.075622] Key type .llcrypt unregistered [ 1439.077899] Key type ._llcrypt unregistered [ 1466.298534] Key type ._llcrypt registered [ 1466.301516] Key type .llcrypt registered [ 1466.426538] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1467.620602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1467.656284] alg: No test for adler32 (adler32-zlib) [ 1469.034979] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 1469.263948] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1471.079960] Key type lgssc registered [ 1472.554788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1479.611960] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1479.643504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1495.264106] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1500.383484] LustreError: 50283:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 1527.071298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1534.096163] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1540.840935] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1708.082478] Lustre: server umount lustre-OST0000 complete [ 1719.996631] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1721.909888] Key type lgssc unregistered [ 1722.113221] LNet: 51957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.120780] LNetError: 51957:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.138244] LNet: Removed LNI 192.168.201.133@tcp [ 1722.742148] Key type .llcrypt unregistered [ 1722.747529] Key type ._llcrypt unregistered [ 1739.039863] Key type ._llcrypt registered [ 1739.041706] Key type .llcrypt registered [ 1739.126801] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1740.024115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1740.044473] alg: No test for adler32 (adler32-zlib) [ 1741.009287] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 1741.228755] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1742.935320] Key type lgssc registered [ 1743.707908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1748.894249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1748.903096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1750.236126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1752.772631] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1758.414404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1758.726329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1762.579830] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1768.239476] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1771.874307] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1776.258428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1776.488761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1780.723452] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1786.963759] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1794.020123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.032188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1795.045290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1800.165254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1800.168691] Lustre: Skipped 1 previous similar message [ 1803.231720] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1803.319872] Lustre: server umount lustre-OST0000 complete [ 1809.891569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1809.900993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.908713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1812.161653] Lustre: server umount lustre-MDT0000 complete [ 1814.700124] LustreError: 53331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783072917 with bad export cookie 17573872536871391743 [ 1814.702675] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1814.707191] LustreError: 53331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1814.939986] Lustre: server umount lustre-MDT0001 complete [ 1822.741320] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1824.592678] Key type lgssc unregistered [ 1824.801961] LNet: 56893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.805236] LNetError: 56893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.816270] LNet: Removed LNI 192.168.201.133@tcp [ 1825.568345] Key type .llcrypt unregistered [ 1825.570810] Key type ._llcrypt unregistered [ 1839.176323] Key type ._llcrypt registered [ 1839.178784] Key type .llcrypt registered [ 1839.238682] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1839.620833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1839.639639] alg: No test for adler32 (adler32-zlib) [ 1840.568614] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 1840.699545] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1842.311164] Key type lgssc registered [ 1842.935299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1847.379973] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1847.399505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1862.879812] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1867.999558] LustreError: 58258:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 1894.623254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.980595] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1904.894593] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1910.249818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.716589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1910.790562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1914.318947] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1915.880372] LustreError: 58284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1915.891237] LustreError: 58284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1921.001196] LustreError: 58283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1921.683723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.971588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1926.244433] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1931.932982] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1936.359544] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1939.426476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.427902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1939.436184] Lustre: Skipped 1 previous similar message [ 1939.445709] Lustre: Skipped 1 previous similar message [ 1944.546106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1949.666259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1949.672309] Lustre: Skipped 2 previous similar messages [ 1952.226979] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1952.333689] Lustre: server umount lustre-OST0000 complete [ 1973.216966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1973.226191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.240291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1975.779081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1976.922563] Lustre: server umount lustre-MDT0000 complete [ 1979.620513] LustreError: 58266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073082 with bad export cookie 9309585676372311582 [ 1979.621410] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1979.628164] LustreError: 58266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1985.883659] Lustre: server umount lustre-MDT0001 complete [ 1992.422215] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1994.247516] Key type lgssc unregistered [ 1994.461283] LNet: 62002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1994.468319] LNetError: 62002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1994.482326] LNet: Removed LNI 192.168.201.133@tcp [ 1995.079763] Key type .llcrypt unregistered [ 1995.081696] Key type ._llcrypt unregistered [ 2011.727219] Key type ._llcrypt registered [ 2011.729924] Key type .llcrypt registered [ 2011.821226] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2012.557275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2012.567726] alg: No test for adler32 (adler32-zlib) [ 2013.476952] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2013.635945] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2015.271225] Key type lgssc registered [ 2015.881731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2020.797907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2020.811723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2022.114226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2024.605062] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2029.702497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2029.968636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2032.526485] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2037.410288] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2040.504581] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2044.016770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2044.236985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2047.339544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2047.749125] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2052.307427] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2067.939199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2067.944182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.952191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2071.012385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2071.015958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2071.017444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.026770] Lustre: Skipped 1 previous similar message [ 2073.497681] Lustre: server umount lustre-OST0000 complete [ 2076.129882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2076.135496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2076.136270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2076.138911] Lustre: Skipped 1 previous similar message [ 2081.252037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2081.255785] Lustre: Skipped 2 previous similar messages [ 2081.671298] Lustre: server umount lustre-MDT0000 complete [ 2084.099878] LustreError: 63373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073186 with bad export cookie 4213482311778443361 [ 2084.101616] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2084.107513] LustreError: 63373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2084.293249] Lustre: server umount lustre-MDT0001 complete [ 2091.061776] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2092.782374] Key type lgssc unregistered [ 2092.951676] LNet: 66947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2092.955803] LNetError: 66947:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.973244] LNet: Removed LNI 192.168.201.133@tcp [ 2093.486199] Key type .llcrypt unregistered [ 2093.487934] Key type ._llcrypt unregistered [ 2107.635327] Key type ._llcrypt registered [ 2107.636697] Key type .llcrypt registered [ 2107.695984] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2108.305668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2108.313231] alg: No test for adler32 (adler32-zlib) [ 2109.221223] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2109.395758] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2111.015189] Key type lgssc registered [ 2111.667073] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2116.945913] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 06:06:58 (1783073218) [ 2121.147334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2121.159804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.526102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2125.149893] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2130.130947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.363563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2132.646664] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2136.366072] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2139.412367] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2143.167397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2143.402868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2146.560618] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2147.496097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2150.499822] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2159.233031] Lustre: Failing over lustre-MDT0000 [ 2159.463408] Lustre: server umount lustre-MDT0000 complete [ 2161.123706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2161.125064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2161.128688] Lustre: Skipped 1 previous similar message [ 2161.129982] LustreError: 68626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.129991] LustreError: 68626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2163.429381] LustreError: 68626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.438055] LustreError: 68626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2165.158859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2165.231703] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2165.306817] Lustre: *** cfs_fail_loc=135, val=0*** [ 2165.356817] LustreError: 71712:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2165.364547] LustreError: 71712:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.133@tcp: cfg command failed: rc = -2 [ 2165.368111] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2165.368111] [ 2165.373880] LustreError: MGC192.168.201.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2165.381044] LustreError: 71700:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2165.386101] LustreError: 71700:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2165.391851] LustreError: 71700:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2165.394222] LustreError: 71700:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2165.406070] Lustre: server umount lustre-MDT0000 complete [ 2165.407788] LustreError: 71700:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2166.242704] LustreError: 68626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.250829] LustreError: 68626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2168.521940] LustreError: 69982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.857684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.936079] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2169.070433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.111546] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2171.066833] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2173.641168] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2174.444173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2174.463820] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2174.495624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2176.481830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2176.486696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.497123] Lustre: Skipped 1 previous similar message [ 2176.502179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.560614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2179.562909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.573043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.526638] Lustre: server umount lustre-OST0000 complete [ 2184.673414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.674103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2184.681454] Lustre: Skipped 1 previous similar message [ 2184.681799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2184.684133] Lustre: Skipped 2 previous similar messages [ 2189.792441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.795892] Lustre: Skipped 1 previous similar message [ 2190.315621] Lustre: server umount lustre-MDT0000 complete [ 2191.986519] LustreError: 68609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073294 with bad export cookie 5187857887173059302 [ 2191.989690] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2191.990901] LustreError: 68609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2192.130534] Lustre: server umount lustre-MDT0001 complete [ 2197.126554] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2198.590657] Key type lgssc unregistered [ 2198.744452] LNet: 73484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2198.748047] LNetError: 73484:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2198.762125] LNet: Removed LNI 192.168.201.133@tcp [ 2199.165443] Key type .llcrypt unregistered [ 2199.166543] Key type ._llcrypt unregistered [ 2210.399357] Key type ._llcrypt registered [ 2210.401020] Key type .llcrypt registered [ 2210.461586] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2210.925425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2210.934922] alg: No test for adler32 (adler32-zlib) [ 2211.886311] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2212.045561] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2213.663747] Key type lgssc registered [ 2214.333411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2218.319385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2218.332639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2219.586983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.062503] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2226.615990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2226.798085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2229.166444] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2233.010079] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2235.581969] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2238.719943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2238.894321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2242.018103] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2245.039828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2245.941716] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2254.575535] Lustre: Failing over lustre-MDT0000 [ 2254.728333] Lustre: server umount lustre-MDT0000 complete [ 2257.377954] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.381961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2257.383442] Lustre: Skipped 2 previous similar messages [ 2257.388747] LustreError: 74876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.401154] LustreError: 74876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2258.651900] LustreError: 74870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.806589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.874098] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2259.968506] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2260.067631] LustreError: 77958:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2260.071327] LustreError: 77958:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.133@tcp: cfg command failed: rc = -9 [ 2260.074646] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2260.074646] [ 2260.079021] LustreError: MGC192.168.201.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2260.084484] LustreError: 77947:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2260.088481] LustreError: 77947:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2260.093603] LustreError: 77947:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2260.096513] LustreError: 77947:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2260.108548] Lustre: server umount lustre-MDT0000 complete [ 2260.110333] LustreError: 77947:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2262.498043] LustreError: 75602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2262.508933] LustreError: 75602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2264.461570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.553252] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2264.781696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.814594] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2267.514157] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2268.875764] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2270.191608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2270.219343] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2270.259361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2272.737061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2272.740361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.746188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2275.298645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2275.298787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.327548] Lustre: server umount lustre-OST0000 complete [ 2285.539361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.541593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2285.549826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2285.559609] Lustre: Skipped 2 previous similar messages [ 2286.933590] Lustre: server umount lustre-MDT0000 complete [ 2289.347461] LustreError: 77450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073391 with bad export cookie 4962459690760011519 [ 2289.356138] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2289.358720] LustreError: 77450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2289.611546] Lustre: server umount lustre-MDT0001 complete [ 2296.673967] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2298.109642] Key type lgssc unregistered [ 2298.276694] LNet: 79735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2298.280863] LNetError: 79735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2298.292666] LNet: Removed LNI 192.168.201.133@tcp [ 2298.798805] Key type .llcrypt unregistered [ 2298.800855] Key type ._llcrypt unregistered [ 2305.177174] Key type ._llcrypt registered [ 2305.178826] Key type .llcrypt registered [ 2305.247177] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2305.442453] Key type .llcrypt unregistered [ 2305.444065] Key type ._llcrypt unregistered [ 2318.682572] Key type ._llcrypt registered [ 2318.687560] Key type .llcrypt registered [ 2318.761436] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2319.237383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2319.286312] alg: No test for adler32 (adler32-zlib) [ 2320.218506] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2320.344072] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2321.943346] Key type lgssc registered [ 2322.536352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2326.536632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2326.550450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2327.756719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.544767] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2334.091818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2334.293894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2336.669918] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2341.187918] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2344.317283] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2348.266287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2348.480565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.941379] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2352.554343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2355.776860] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2367.970426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2367.975575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.984356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2368.993413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2368.993766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2369.006259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.071650] Lustre: server umount lustre-OST0000 complete [ 2374.114967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.119163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2374.121725] Lustre: Skipped 1 previous similar message [ 2374.132688] Lustre: Skipped 1 previous similar message [ 2379.167450] Lustre: server umount lustre-MDT0000 complete [ 2379.236889] LustreError: 81614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.247871] LustreError: 81614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2381.519136] LustreError: 81596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073484 with bad export cookie 1472100220891379236 [ 2381.522086] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2381.525404] LustreError: 81596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2381.775331] Lustre: server umount lustre-MDT0001 complete [ 2388.510713] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2390.113357] Key type lgssc unregistered [ 2390.306741] LNet: 85143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.312857] LNetError: 85143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2390.331334] LNet: Removed LNI 192.168.201.133@tcp [ 2390.779336] Key type .llcrypt unregistered [ 2390.781128] Key type ._llcrypt unregistered [ 2403.737699] Key type ._llcrypt registered [ 2403.739990] Key type .llcrypt registered [ 2403.802576] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2404.290733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2404.330780] alg: No test for adler32 (adler32-zlib) [ 2405.216915] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2405.362783] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2406.967138] Key type lgssc registered [ 2407.497681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2411.436764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2411.448654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2412.679673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.958758] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2419.502414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2419.771046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2421.767380] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2425.556470] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2428.260837] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2431.819700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2432.028314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.993241] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2438.189642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2438.798417] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2448.353569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2448.358421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.366497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2450.399980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2450.403803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2450.408227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2453.790813] Lustre: server umount lustre-OST0000 complete [ 2460.644201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2460.645539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.654357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2460.654365] Lustre: Skipped 1 previous similar message [ 2460.675148] Lustre: Skipped 1 previous similar message [ 2461.800155] Lustre: server umount lustre-MDT0000 complete [ 2463.912661] LustreError: 86514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073566 with bad export cookie 11529351240164132181 [ 2463.915278] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2463.923679] LustreError: 86514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2464.120519] Lustre: server umount lustre-MDT0001 complete [ 2468.930796] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2470.201312] Key type lgssc unregistered [ 2470.372547] LNet: 90062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2470.376774] LNetError: 90062:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2470.395673] LNet: Removed LNI 192.168.201.133@tcp [ 2470.780076] Key type .llcrypt unregistered [ 2470.783932] Key type ._llcrypt unregistered [ 2481.484478] Key type ._llcrypt registered [ 2481.485888] Key type .llcrypt registered [ 2481.529168] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2481.968469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2481.992564] alg: No test for adler32 (adler32-zlib) [ 2482.869919] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2482.970529] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2484.567177] Key type lgssc registered [ 2485.011761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2488.171901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2488.182422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2489.367125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2490.957743] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2494.334124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2494.492416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2496.187679] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2499.167277] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2501.372420] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2504.163030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2504.302250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.932274] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2509.997766] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2513.508842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2524.128955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2524.130592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.133670] LustreError: Skipped 1 previous similar message [ 2524.138905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2529.249529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2529.252572] Lustre: Skipped 2 previous similar messages [ 2529.941514] Lustre: server umount lustre-OST0000 complete [ 2534.369686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.372325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2534.374990] Lustre: Skipped 2 previous similar messages [ 2537.630854] Lustre: server umount lustre-MDT0000 complete [ 2539.162882] LustreError: 91433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073641 with bad export cookie 6962757315698448377 [ 2539.163755] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2539.166481] LustreError: 91433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2539.289966] Lustre: server umount lustre-MDT0001 complete [ 2543.494616] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2544.733623] Key type lgssc unregistered [ 2544.882437] LNet: 95002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2544.885924] LNetError: 95002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2544.904595] LNet: Removed LNI 192.168.201.133@tcp [ 2545.350271] Key type .llcrypt unregistered [ 2545.351412] Key type ._llcrypt unregistered [ 2556.636740] Key type ._llcrypt registered [ 2556.637979] Key type .llcrypt registered [ 2556.686305] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2557.147906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2557.157835] alg: No test for adler32 (adler32-zlib) [ 2558.032976] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2558.139152] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2559.727226] Key type lgssc registered [ 2560.173259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2563.437672] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2563.447134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2578.911396] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2584.031259] LustreError: 96379:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 2610.655289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2612.982384] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2615.904894] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2618.408328] Lustre: server umount lustre-OST0000 complete [ 2621.972619] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 06:15:23 (1783073723) [ 2624.752496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.984963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2626.758413] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2630.112120] LustreError: 98015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.117801] LustreError: 98015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2630.127389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2630.290670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2631.995762] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2635.014906] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2637.275613] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2639.904263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2640.059960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2642.236878] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2645.165270] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2650.282742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2660.832561] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2660.833847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.835683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2660.836051] LustreError: Skipped 1 previous similar message [ 2660.840611] Lustre: Skipped 1 previous similar message [ 2662.488986] Lustre: server umount lustre-OST0000 complete [ 2665.953547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2665.954665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2665.957647] Lustre: Skipped 1 previous similar message [ 2665.969947] Lustre: Skipped 2 previous similar messages [ 2666.219181] Lustre: server umount lustre-MDT0000 complete [ 2667.768237] LustreError: 98752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073770 with bad export cookie 17609723049183164009 [ 2667.770774] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2667.772570] LustreError: 98752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2667.908074] Lustre: server umount lustre-MDT0001 complete [ 2672.161881] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2673.348469] Key type lgssc unregistered [ 2673.495810] LNet: 101722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2673.499247] LNetError: 101722:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2673.508330] LNet: Removed LNI 192.168.201.133@tcp [ 2673.858611] Key type .llcrypt unregistered [ 2673.859659] Key type ._llcrypt unregistered [ 2690.965222] Key type ._llcrypt registered [ 2690.966716] Key type .llcrypt registered [ 2691.022791] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2691.444168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2691.459890] alg: No test for adler32 (adler32-zlib) [ 2692.370365] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2692.475811] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2694.071182] Key type lgssc registered [ 2694.513283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2697.817964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2697.833604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.035728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2700.717868] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2704.553860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.738717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2706.441150] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2709.352200] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2711.418242] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2714.012738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2714.142498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.389346] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2719.458161] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2724.391761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2729.443469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2729.444517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.448741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2729.453028] Lustre: Skipped 1 previous similar message [ 2733.802661] Lustre: server umount lustre-OST0000 complete [ 2735.583833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2735.586757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.592579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2735.595263] Lustre: Skipped 1 previous similar message [ 2739.681252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.681397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.547901] Lustre: server umount lustre-MDT0000 complete [ 2742.951617] LustreError: 104616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073845 with bad export cookie 11867509577065115615 [ 2742.953658] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2742.958046] LustreError: 104616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2743.094115] Lustre: server umount lustre-MDT0001 complete [ 2747.202925] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2748.447979] Key type lgssc unregistered [ 2748.619591] LNet: 107413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2748.623344] LNetError: 107413:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2748.637355] LNet: Removed LNI 192.168.201.133@tcp [ 2749.050371] Key type .llcrypt unregistered [ 2749.051570] Key type ._llcrypt unregistered [ 2757.337964] Key type ._llcrypt registered [ 2757.339176] Key type .llcrypt registered [ 2757.389211] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2757.855429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2757.878702] alg: No test for adler32 (adler32-zlib) [ 2758.760452] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2758.877653] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2760.471212] Key type lgssc registered [ 2760.960637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2764.184613] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2764.191263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2779.615318] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2784.735298] LustreError: 108632:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 2811.359186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2813.531261] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2816.352435] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2818.994732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.102697] LustreError: MGC192.168.201.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2819.109170] LustreError: 109659:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2819.112417] LustreError: 109659:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2819.116401] LustreError: 109659:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2819.148661] Lustre: server umount lustre-MDT0000 complete [ 2819.150077] LustreError: 109659:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2832.393967] Lustre: server umount lustre-OST0000 complete [ 2835.872499] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 2839.400856] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2852.438267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.534865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2852.547918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2852.586667] Lustre: lustre-MDT0000: new disk, initializing [ 2852.629250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.637796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2854.162468] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2858.729567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2858.775053] Lustre: 112445:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2858.805189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2858.807805] Lustre: Skipped 1 previous similar message [ 2858.847675] Lustre: lustre-MDT0001: new disk, initializing [ 2858.897362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2858.913335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2858.918233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2860.393869] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2864.333978] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2866.312723] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2868.879066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2869.009094] Lustre: lustre-OST0000: new disk, initializing [ 2869.011581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2869.014780] Lustre: 114415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2869.059818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2871.272823] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2875.351302] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2876.424166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2876.429269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2876.472742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2877.034366] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2877.136099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2878.720151] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2878.816920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2881.505698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2881.506765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2881.511231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2881.514224] LustreError: Skipped 1 previous similar message [ 2881.516969] Lustre: Skipped 1 previous similar message [ 2885.723257] Lustre: server umount lustre-OST0000 complete [ 2889.697688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2889.701688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2889.707485] Lustre: Skipped 1 previous similar message [ 2889.710354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.744144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.744733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.339544] Lustre: server umount lustre-MDT0000 complete [ 2894.989227] LustreError: 113241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073997 with bad export cookie 14835828811115465131 [ 2894.992590] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2894.992970] LustreError: 113241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2895.130765] Lustre: server umount lustre-MDT0001 complete [ 2898.424687] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 06:20:00 (1783074000) [ 2899.890471] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2906.230144] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 2910.025197] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2924.718478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2924.812432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2924.830966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2924.871400] Lustre: lustre-MDT0000: new disk, initializing [ 2924.912963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.922876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2926.616517] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2931.515510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.571915] Lustre: 119304:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2931.598654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2931.600740] Lustre: Skipped 1 previous similar message [ 2931.638117] Lustre: lustre-MDT0001: new disk, initializing [ 2931.693152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2931.696943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2933.457731] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2937.693297] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2939.961233] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2942.491395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2942.602616] Lustre: lustre-OST0000: new disk, initializing [ 2942.604921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2942.609066] Lustre: 121274:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2942.612423] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2942.614773] Lustre: Skipped 1 previous similar message [ 2942.642603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2942.644631] Lustre: Skipped 1 previous similar message [ 2944.940578] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2948.096961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2948.101344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2948.121403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2949.272092] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2950.950434] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2951.036101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2952.766050] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2952.854252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2954.725086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2954.725603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.726771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2954.735357] Lustre: Skipped 1 previous similar message [ 2955.804355] Lustre: server umount lustre-OST0000 complete [ 2959.841249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.847901] Lustre: Skipped 1 previous similar message [ 2963.550371] Lustre: server umount lustre-MDT0000 complete [ 2964.983395] LustreError: 119296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074067 with bad export cookie 14835828811115465999 [ 2964.985916] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2964.987280] LustreError: 119296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2965.112240] Lustre: server umount lustre-MDT0001 complete [ 2968.547407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2968.857289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2970.640926] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2974.080444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.175968] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2974.179469] Lustre: Skipped 3 previous similar messages [ 2975.748205] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2978.782019] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2980.722831] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2983.520177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2985.862073] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2988.615397] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3001.790796] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3005.411433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.412539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3005.418631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3005.420050] LustreError: Skipped 1 previous similar message [ 3005.425227] Lustre: Skipped 1 previous similar message [ 3009.064269] Lustre: server umount lustre-OST0000 complete [ 3015.137459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3016.767213] Lustre: server umount lustre-MDT0000 complete [ 3018.386544] LustreError: 123465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074120 with bad export cookie 14835828811115466860 [ 3018.388625] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3018.389662] LustreError: 123465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3018.517575] Lustre: server umount lustre-MDT0001 complete [ 3022.541171] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3023.683350] Key type lgssc unregistered [ 3023.818470] LNet: 127134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3023.821111] LNetError: 127134:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3023.831668] LNet: Removed LNI 192.168.201.133@tcp [ 3024.190363] Key type .llcrypt unregistered [ 3024.191493] Key type ._llcrypt unregistered [ 3030.776220] Key type ._llcrypt registered [ 3030.777276] Key type .llcrypt registered [ 3030.815943] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 3037.696305] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3038.217852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3038.242086] alg: No test for adler32 (adler32-zlib) [ 3039.157750] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 3039.277631] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3040.887136] Key type lgssc registered [ 3041.450152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3055.209490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3055.215236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3056.302927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3056.314202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3056.348250] Lustre: lustre-MDT0000: new disk, initializing [ 3056.369386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3056.375897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3057.862962] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3062.467192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.515373] Lustre: 130260:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3062.539976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3062.541844] Lustre: Skipped 1 previous similar message [ 3062.580116] Lustre: lustre-MDT0001: new disk, initializing [ 3062.624665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3062.639234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3062.646814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3064.178694] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3068.392330] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3070.405919] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3072.937580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3073.072983] Lustre: lustre-OST0000: new disk, initializing [ 3073.075405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3073.079748] Lustre: 132225:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3073.120301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.208605] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3079.286851] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3080.833924] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3082.232851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3082.236217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3082.275301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3082.946866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3084.393132] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3084.486989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3087.329832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3087.331035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3087.334949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3087.336990] LustreError: Skipped 1 previous similar message [ 3087.339171] Lustre: Skipped 1 previous similar message [ 3091.347698] Lustre: server umount lustre-OST0000 complete [ 3093.471830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3093.474244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3093.477864] Lustre: Skipped 1 previous similar message [ 3093.479583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3097.568073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3097.568543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3102.688252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3102.690677] Lustre: Skipped 2 previous similar messages [ 3106.784616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3106.787204] Lustre: Skipped 1 previous similar message [ 3107.295158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3107.372403] Lustre: server umount lustre-MDT0000 complete [ 3108.851261] LustreError: 133781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074211 with bad export cookie 18313732225200742989 [ 3108.854183] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3108.855598] LustreError: 133781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3108.975173] Lustre: server umount lustre-MDT0001 complete [ 3112.161214] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 06:23:33 (1783074213) [ 3114.673376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.875477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.494741] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3120.069382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.096064] LustreError: 134652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3120.102813] LustreError: 134652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3120.220671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3121.824719] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3124.630529] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3126.756857] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3129.569215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.570027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3129.574808] Lustre: Skipped 1 previous similar message [ 3133.667350] Lustre: server umount lustre-MDT0000 complete [ 3134.944075] LustreError: 134648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3134.949709] LustreError: 134648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3135.143403] LustreError: 134632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074237 with bad export cookie 18313732225200743857 [ 3135.145480] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3135.148297] LustreError: 134632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3135.268163] Lustre: server umount lustre-MDT0001 complete [ 3138.439842] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 06:24:00 (1783074240) [ 3141.150961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3156.575589] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3161.695553] LustreError: 137063:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 3187.295320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3189.184640] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3191.520375] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3192.249088] Lustre: server umount lustre-OST0000 complete [ 3194.818823] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 06:24:56 (1783074296) [ 3196.972211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.114453] LustreError: 138506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.499348] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3201.509047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3203.078304] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3205.640351] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3207.537058] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3210.039639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3210.161256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3210.163746] Lustre: Skipped 2 previous similar messages [ 3212.219863] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3214.751990] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3227.103849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3227.106921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.111961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3227.114890] Lustre: Skipped 1 previous similar message [ 3233.119831] Lustre: server umount lustre-MDT0000 complete [ 3234.529507] LustreError: 138488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074337 with bad export cookie 18313732225200744508 [ 3234.532321] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3234.533837] LustreError: 138488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3234.650618] Lustre: server umount lustre-MDT0001 complete [ 3246.304934] Lustre: server umount lustre-OST0000 complete [ 3249.073039] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 06:25:51 (1783074351) [ 3251.387378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3251.575319] LustreError: 142177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.603173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.001773] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3256.055887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.660139] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3260.116925] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3262.003858] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3264.453277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3266.707260] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3269.213322] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3270.673705] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3276.898953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3277.899610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3279.346844] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3279.426878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3281.375722] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3281.378265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3281.381987] Lustre: Skipped 2 previous similar messages [ 3281.383773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3281.385958] Lustre: Skipped 5 previous similar messages [ 3286.234724] Lustre: server umount lustre-OST0000 complete [ 3292.127809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3292.130616] LustreError: Skipped 1 previous similar message [ 3293.717320] Lustre: server umount lustre-MDT0000 complete [ 3294.936933] LustreError: 144018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074397 with bad export cookie 18313732225200745565 [ 3294.940074] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3294.940904] LustreError: 144018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3295.049256] Lustre: server umount lustre-MDT0001 complete [ 3297.604129] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 06:26:39 (1783074399) [ 3299.850702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3315.295535] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3320.415406] LustreError: 146379:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 3346.015320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3346.016838] Lustre: Skipped 2 previous similar messages [ 3348.013388] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3350.566801] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3352.957243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3353.117536] LustreError: 146404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.167946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3354.603451] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3357.689166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.219376] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3361.677525] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3363.467102] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3365.050450] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3365.142511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3366.651892] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3366.732114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3368.416260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3368.419073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.423995] Lustre: Skipped 3 previous similar messages [ 3368.425738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3368.427568] Lustre: Skipped 4 previous similar messages [ 3373.520668] Lustre: server umount lustre-MDT0000 complete [ 3374.752613] LustreError: 146387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074477 with bad export cookie 18313732225200746573 [ 3374.754061] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3374.756144] LustreError: 146387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3374.860418] Lustre: server umount lustre-MDT0001 complete [ 3386.325260] Lustre: server umount lustre-OST0000 complete [ 3389.289687] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 06:28:11 (1783074491) [ 3391.581740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3407.007438] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3412.127473] LustreError: 150572:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 3439.762760] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3442.220096] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3444.517801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3444.659667] LustreError: 150597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3444.709413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3446.054229] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3448.922744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3450.451678] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3452.912690] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3454.767612] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3457.123540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3457.192947] Lustre: lustre-OST0001: new disk, initializing [ 3457.195287] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3457.198734] Lustre: 153438:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3457.629054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3457.633470] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3457.661827] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3459.130632] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3462.819382] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3464.307254] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3464.377424] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3465.685253] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3465.753040] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3468.259485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.262505] Lustre: Skipped 2 previous similar messages [ 3472.445621] Lustre: server umount lustre-OST0000 complete [ 3473.380359] LustreError: 151630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3473.385901] LustreError: 151630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3478.496968] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3488.223101] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3488.278476] Lustre: server umount lustre-OST0001 complete [ 3495.649655] Lustre: server umount lustre-MDT0000 complete [ 3496.919072] LustreError: 150580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074599 with bad export cookie 18313732225200747525 [ 3496.922104] LustreError: 150580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3496.922141] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3497.044676] Lustre: server umount lustre-MDT0001 complete [ 3504.727401] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 06:30:06 (1783074606) [ 3505.285167] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3505.799346] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3506.321315] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 06:30:08 (1783074608) [ 3508.472472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3508.562096] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3508.572037] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3508.642740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.644883] Lustre: Skipped 6 previous similar messages [ 3509.971962] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3512.708649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3512.747591] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3512.756626] Lustre: 156995:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3514.145550] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3516.563955] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3518.355148] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3520.662730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3520.728343] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3522.708522] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3524.882038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3525.085673] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3526.461212] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3526.532798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3527.926934] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3528.011992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3530.209301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3530.210902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3530.211427] LustreError: Skipped 3 previous similar messages [ 3530.213222] Lustre: Skipped 12 previous similar messages [ 3534.804990] Lustre: server umount lustre-OST0000 complete [ 3548.513923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3549.668444] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3549.673169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3549.674355] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3549.680571] Lustre: Skipped 1 previous similar message [ 3550.566109] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3552.844200] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3554.335508] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3554.411597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3555.763447] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3555.840596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3557.589414] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8f2544375000.ost_server_uuid 50 [ 3558.078989] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f2544375000.ost_server_uuid in FULL state after 0 sec [ 3558.139378] LustreError: 157000:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.33@tcp) returned error from blocking AST (req@00000000cf263d10 x1869688750663424 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000d62c8c25/0xfe27726db2b6fd51 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.33@tcp remote: 0xccb85c6f404318d7 expref: 5 pid: 157001 timeout: 3658 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3558.151888] LustreError: lustre-MDT0000: A client on nid 192.168.201.33@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3558.155449] LustreError: 156990:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.33@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000d62c8c25/0xfe27726db2b6fd51 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.33@tcp remote: 0xccb85c6f404318d7 expref: 6 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3559.905857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3559.909521] Lustre: Skipped 8 previous similar messages [ 3565.260565] Lustre: server umount lustre-OST0000 complete [ 3572.627806] Lustre: server umount lustre-MDT0000 complete [ 3573.839119] LustreError: 162057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074676 with bad export cookie 18313732225200748708 [ 3573.841271] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3573.843092] LustreError: 162057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3580.016680] Lustre: server umount lustre-MDT0001 complete [ 3583.809332] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3584.868756] Key type lgssc unregistered [ 3585.004518] LNet: 163096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3585.011780] LNetError: 163096:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3585.027385] LNet: Removed LNI 192.168.201.133@tcp [ 3585.381653] Key type .llcrypt unregistered [ 3585.383112] Key type ._llcrypt unregistered [ 3594.896989] Key type ._llcrypt registered [ 3594.897957] Key type .llcrypt registered [ 3594.942448] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3595.355073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3595.368640] alg: No test for adler32 (adler32-zlib) [ 3596.245150] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 3596.339649] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3597.919220] Key type lgssc registered [ 3598.304800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3601.162107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3601.182576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3602.381076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3603.898435] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3607.086096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.237776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3608.720612] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3611.369771] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3613.318400] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3615.678733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3615.800800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3617.857266] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3620.290966] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3627.045562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3633.990124] Lustre: Failing over lustre-MDT0000 [ 3634.200396] Lustre: server umount lustre-MDT0000 complete [ 3637.217041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3637.218951] LustreError: 166982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3637.223094] Lustre: Skipped 2 previous similar messages [ 3637.229672] LustreError: 166982:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3642.336058] LustreError: 164485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.347030] LustreError: 164485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3645.071349] LustreError: 166326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.33@tcp arrived at 1783074747 with bad export cookie 13112430000464729463 [ 3645.076162] LustreError: 166326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3647.457027] LustreError: 164486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3647.461769] LustreError: 164486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3647.584337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3647.624787] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3647.725870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3647.756715] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3649.183374] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3649.974720] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3649.977937] Lustre: lustre-MDT0000: Denying connection for new client 0aca4e88-2dc0-45d9-b926-aeadb5e26d85 (at 192.168.201.33@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3653.092813] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3657.695987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3657.703574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.715780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3663.329238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3663.331558] Lustre: Skipped 3 previous similar messages [ 3668.449929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3668.452799] Lustre: Skipped 1 previous similar message [ 3671.007112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3671.077301] Lustre: server umount lustre-OST0000 complete [ 3672.396545] LustreError: 168234:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3672.399196] Lustre: 167583:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3672.403142] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3672.409540] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3672.414463] LustreError: 167583:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002343e124 x1869689334568064/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3672.420188] LustreError: 167583:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3672.425057] LustreError: 167583:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3672.429733] Lustre: lustre-MDT0000: Recovery over after 0:23, of 2 clients 0 recovered and 2 were evicted. [ 3672.526453] Lustre: server umount lustre-MDT0000 complete [ 3673.568231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.573470] Lustre: Skipped 1 previous similar message [ 3673.575535] LustreError: 164481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3673.871681] LustreError: 164512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074776 with bad export cookie 13112430000464729792 [ 3673.872813] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3673.876127] LustreError: 164512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3680.050608] Lustre: server umount lustre-MDT0001 complete [ 3683.491146] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3684.503248] Key type lgssc unregistered [ 3684.632464] LNet: 168987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3684.636032] LNetError: 168987:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3684.642315] LNet: Removed LNI 192.168.201.133@tcp [ 3684.966385] Key type .llcrypt unregistered [ 3684.967975] Key type ._llcrypt unregistered [ 3693.437217] Key type ._llcrypt registered [ 3693.438237] Key type .llcrypt registered [ 3693.471301] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3693.822384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3693.830939] alg: No test for adler32 (adler32-zlib) [ 3694.673491] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 3694.744618] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3696.319128] Key type lgssc registered [ 3696.637026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3699.220179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3699.227440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3700.359886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3701.666523] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3704.340200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.461796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3705.721314] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3708.016124] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3709.895326] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3712.012935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3712.118459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.035777] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3716.249082] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3725.413386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3729.887846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3729.890219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3729.896466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3730.912623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3730.915045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3730.915052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3733.834698] Lustre: server umount lustre-OST0000 complete [ 3735.519851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3735.522787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3735.527799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3735.529885] Lustre: Skipped 1 previous similar message [ 3741.152463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3741.154120] Lustre: Skipped 3 previous similar messages [ 3746.272235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3746.273826] Lustre: Skipped 1 previous similar message [ 3749.343081] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3749.406907] Lustre: server umount lustre-MDT0000 complete [ 3750.634249] LustreError: 170356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074853 with bad export cookie 14605277203881219056 [ 3750.637381] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3750.638443] LustreError: 170356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3750.764131] Lustre: server umount lustre-MDT0001 complete [ 3754.311402] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3755.299177] Key type lgssc unregistered [ 3755.434369] LNet: 173925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3755.437902] LNetError: 173925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3755.446659] LNet: Removed LNI 192.168.201.133@tcp [ 3755.755468] Key type .llcrypt unregistered [ 3755.757064] Key type ._llcrypt unregistered [ 3767.995034] Key type ._llcrypt registered [ 3767.996053] Key type .llcrypt registered [ 3768.031864] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3768.465928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3768.477851] alg: No test for adler32 (adler32-zlib) [ 3769.339568] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 3769.433696] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3771.015155] Key type lgssc registered [ 3771.355246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3773.881589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3773.888925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3775.040300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3776.578636] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3779.587124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3779.745550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3781.299178] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3784.039569] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3785.986247] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3788.454116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3788.579174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3790.596928] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3793.068730] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3799.847841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3805.154123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3805.155166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3805.159425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3805.162785] LustreError: Skipped 1 previous similar message [ 3806.803637] Lustre: server umount lustre-OST0000 complete [ 3810.196042] Lustre: server umount lustre-MDT0000 complete [ 3810.272923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.277742] Lustre: Skipped 1 previous similar message [ 3810.279737] LustreError: 175597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.286722] LustreError: 175597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3811.431259] LustreError: 175582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074913 with bad export cookie 7228394504365062165 [ 3811.433035] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3811.435676] LustreError: 175582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3811.552414] Lustre: server umount lustre-MDT0001 complete [ 3815.327628] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3816.458672] Key type lgssc unregistered [ 3816.586383] LNet: 179147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3816.588561] LNetError: 179147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3816.597443] LNet: Removed LNI 192.168.201.133@tcp [ 3816.900975] Key type .llcrypt unregistered [ 3816.902646] Key type ._llcrypt unregistered [ 3825.756693] Key type ._llcrypt registered [ 3825.758216] Key type .llcrypt registered [ 3825.799389] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3826.119986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3826.159053] alg: No test for adler32 (adler32-zlib) [ 3827.020576] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 3827.106587] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3828.687135] Key type lgssc registered [ 3829.050790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3832.641326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3832.646779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3833.765445] Lustre: *** cfs_fail_loc=135, val=0*** [ 3833.800462] LustreError: 180646:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3833.802357] LustreError: 180646:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.133@tcp: cfg command failed: rc = -2 [ 3833.804904] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3833.804904] [ 3833.807875] LustreError: MGC192.168.201.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3833.812672] LustreError: 180608:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3833.815527] LustreError: 180608:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3833.818100] LustreError: 180608:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3833.820450] LustreError: 180608:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3833.852853] Lustre: server umount lustre-MDT0000 complete [ 3833.854681] LustreError: 180608:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3836.094080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3836.234562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3837.625619] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3840.427022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3840.548603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3841.909715] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3844.174850] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3846.077465] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3850.719892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3850.722844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.728145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3851.744773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3851.746075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3852.884546] Lustre: server umount lustre-MDT0000 complete [ 3854.128094] LustreError: 181001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074956 with bad export cookie 2196189761237015945 [ 3854.130748] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3854.134118] LustreError: 181001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3854.236169] Lustre: server umount lustre-MDT0001 complete [ 3857.754749] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3858.807699] Key type lgssc unregistered [ 3858.947356] LNet: 183338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3858.949873] LNetError: 183338:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3858.958794] LNet: Removed LNI 192.168.201.133@tcp [ 3859.258839] Key type .llcrypt unregistered [ 3859.260090] Key type ._llcrypt unregistered [ 3866.039390] Key type ._llcrypt registered [ 3866.040849] Key type .llcrypt registered [ 3866.079885] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3866.189363] Key type .llcrypt unregistered [ 3866.190399] Key type ._llcrypt unregistered [ 3873.009142] Key type ._llcrypt registered [ 3873.010904] Key type .llcrypt registered [ 3873.047202] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3873.335286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3873.344532] alg: No test for adler32 (adler32-zlib) [ 3874.198359] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 3874.289157] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3875.871137] Key type lgssc registered [ 3876.288651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3879.127281] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3879.135558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3894.559578] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3899.679397] LustreError: 185450:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 3926.303205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3928.296940] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3930.708580] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3933.000717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3933.166788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3933.194779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3934.467662] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3937.241438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3937.366202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3938.749370] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3941.089257] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3942.872400] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3943.809080] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3952.097604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3952.099086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3952.101532] Lustre: Skipped 1 previous similar message [ 3957.129413] Lustre: server umount lustre-OST0000 complete [ 3962.336766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3962.337344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3962.339792] Lustre: Skipped 1 previous similar message [ 3962.341659] Lustre: Skipped 1 previous similar message [ 3964.434096] Lustre: server umount lustre-MDT0000 complete [ 3965.625340] LustreError: 188120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075068 with bad export cookie 14322370537716868317 [ 3965.627261] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3965.628475] LustreError: 188120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3965.740715] Lustre: server umount lustre-MDT0001 complete [ 3969.229935] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3970.196370] Key type lgssc unregistered [ 3970.307348] LNet: 189468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3970.310552] LNetError: 189468:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3970.324266] LNet: Removed LNI 192.168.201.133@tcp [ 3970.617572] Key type .llcrypt unregistered [ 3970.618852] Key type ._llcrypt unregistered [ 3979.069768] Key type ._llcrypt registered [ 3979.070741] Key type .llcrypt registered [ 3979.101697] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3979.393877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3979.418111] alg: No test for adler32 (adler32-zlib) [ 3980.259110] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 3980.334727] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3981.911172] Key type lgssc registered [ 3982.269856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3984.803905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3984.809132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.940301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3987.225441] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3989.977155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3990.107974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3991.406208] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3993.615448] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3995.344613] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3997.538385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3997.670601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3999.528246] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4001.847437] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4010.983307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4013.296197] Lustre: Failing over lustre-MDT0000 [ 4013.384389] Lustre: server umount lustre-MDT0000 complete [ 4015.583809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4015.585846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.589433] LustreError: 193346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4015.595038] LustreError: 193346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4016.096721] LustreError: 190857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4016.097125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4016.101908] LustreError: 190857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4016.106122] Lustre: Skipped 1 previous similar message [ 4017.360358] LustreError: 190853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4017.367858] LustreError: 190853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4021.216036] LustreError: 191586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.220240] LustreError: 191586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4026.036187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4026.076876] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4026.166828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4026.192627] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4027.540664] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4027.593251] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4029.096097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4031.460714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4031.467678] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4031.485072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4036.249462] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4045.791987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4045.795549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.801187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4046.818070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4046.818912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4046.823897] Lustre: Skipped 1 previous similar message [ 4049.557347] Lustre: server umount lustre-OST0000 complete [ 4051.938863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4051.939032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.939598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4051.947473] Lustre: Skipped 2 previous similar messages [ 4057.059990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4057.063285] Lustre: Skipped 2 previous similar messages [ 4062.176539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4065.247151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4065.324403] Lustre: server umount lustre-MDT0000 complete [ 4066.510237] LustreError: 193507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075169 with bad export cookie 13877421689670122476 [ 4066.512933] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4066.514808] LustreError: 193507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4066.628160] Lustre: server umount lustre-MDT0001 complete [ 4069.962520] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4070.935522] Key type lgssc unregistered [ 4071.058325] LNet: 195850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4071.060882] LNetError: 195850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4071.069312] LNet: Removed LNI 192.168.201.133@tcp [ 4071.356917] Key type .llcrypt unregistered [ 4071.357676] Key type ._llcrypt unregistered [ 4079.965333] Key type ._llcrypt registered [ 4079.966205] Key type .llcrypt registered [ 4080.002863] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4080.403656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4080.421306] alg: No test for adler32 (adler32-zlib) [ 4081.261567] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 4081.348163] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4082.927160] Key type lgssc registered [ 4083.263526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4085.733589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4085.740776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4086.894744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4088.268449] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4091.066985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4091.201641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4092.527405] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4094.925382] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4096.775241] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4099.042888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4099.152774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4101.020451] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4103.329167] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4113.510874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4114.197715] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4116.641454] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4126.109284] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4132.321057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4132.323826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4132.328930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4134.372195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.372639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4134.377803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4134.378961] Lustre: Skipped 1 previous similar message [ 4137.295265] Lustre: server umount lustre-OST0000 complete [ 4139.488644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4139.489263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4139.493432] Lustre: Skipped 1 previous similar message [ 4144.608559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4144.610338] Lustre: Skipped 2 previous similar messages [ 4144.661979] Lustre: server umount lustre-MDT0000 complete [ 4145.827088] LustreError: 197220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075248 with bad export cookie 9149273218832360534 [ 4145.829579] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4145.830834] LustreError: 197220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4145.949828] Lustre: server umount lustre-MDT0001 complete [ 4149.349141] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4150.328259] Key type lgssc unregistered [ 4150.456417] LNet: 201102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4150.459986] LNetError: 201102:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4150.468250] LNet: Removed LNI 192.168.201.133@tcp [ 4150.755285] Key type .llcrypt unregistered [ 4150.756150] Key type ._llcrypt unregistered [ 4157.886833] Key type ._llcrypt registered [ 4157.887750] Key type .llcrypt registered [ 4157.926064] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4158.260321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4158.287081] alg: No test for adler32 (adler32-zlib) [ 4159.138759] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 4159.225653] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4160.807148] Key type lgssc registered [ 4161.162935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4163.589458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4163.595365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4164.724930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4165.956798] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4168.626265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4168.743350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4169.974200] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4172.191298] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4173.863610] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4176.039046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4176.156598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4178.048985] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4180.393678] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4190.501601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4192.332941] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4202.564474] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4208.014333] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4212.985735] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4220.384108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4220.387029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4220.392251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4221.411453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4221.412344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4221.412658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4225.350412] Lustre: server umount lustre-OST0000 complete [ 4226.528283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4226.528824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4226.531158] Lustre: Skipped 1 previous similar message [ 4226.533111] Lustre: Skipped 1 previous similar message [ 4231.648439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4231.650583] Lustre: Skipped 1 previous similar message [ 4232.663792] Lustre: server umount lustre-MDT0000 complete [ 4233.896223] LustreError: 202369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075336 with bad export cookie 1662411444565828525 [ 4233.899439] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4233.900718] LustreError: 202369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4234.003353] Lustre: server umount lustre-MDT0001 complete [ 4237.443398] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4238.481926] Key type lgssc unregistered [ 4238.601363] LNet: 207620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4238.603669] LNetError: 207620:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4238.615272] LNet: Removed LNI 192.168.201.133@tcp [ 4238.906297] Key type .llcrypt unregistered [ 4238.907173] Key type ._llcrypt unregistered [ 4247.463961] Key type ._llcrypt registered [ 4247.465193] Key type .llcrypt registered [ 4247.500182] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4247.808479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4247.817820] alg: No test for adler32 (adler32-zlib) [ 4248.676851] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 4248.763033] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4250.351150] Key type lgssc registered [ 4250.706315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4253.166961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4253.172738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4254.299381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4255.599511] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4258.409625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4258.531167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4259.836718] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4262.127187] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4263.980233] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4266.254533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4266.377590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4268.425344] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4270.886278] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4278.695547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4284.895984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4284.899300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4284.904049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4288.994427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4288.994433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4288.994494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4294.115909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4294.117850] Lustre: Skipped 1 previous similar message [ 4299.231129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4299.234680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4299.236697] Lustre: Skipped 1 previous similar message [ 4299.287488] Lustre: server umount lustre-OST0000 complete [ 4304.352484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4304.353250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4304.357222] Lustre: Skipped 1 previous similar message [ 4304.359571] Lustre: Skipped 1 previous similar message [ 4306.643266] Lustre: server umount lustre-MDT0000 complete [ 4307.892453] LustreError: 208990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075410 with bad export cookie 7342379353640169351 [ 4307.895185] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4307.897056] LustreError: 208990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4308.010238] Lustre: server umount lustre-MDT0001 complete [ 4311.314823] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4312.233617] Key type lgssc unregistered [ 4312.363436] LNet: 212711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4312.365780] LNetError: 212711:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4312.377455] LNet: Removed LNI 192.168.201.133@tcp [ 4312.672383] Key type .llcrypt unregistered [ 4312.673647] Key type ._llcrypt unregistered [ 4319.513026] Key type ._llcrypt registered [ 4319.514607] Key type .llcrypt registered [ 4319.561097] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4319.972420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4320.016195] alg: No test for adler32 (adler32-zlib) [ 4320.872072] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 4320.954845] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4322.551103] Key type lgssc registered [ 4322.891396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4325.339453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4325.345029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4326.483044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4327.746876] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4330.389919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4330.514927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4331.876849] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4334.133719] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4335.796355] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4338.047275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4338.176413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4340.246121] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4342.586959] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4351.525100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4356.577126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4356.577226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4356.577875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4356.580527] Lustre: Skipped 1 previous similar message [ 4358.420141] Lustre: server umount lustre-OST0000 complete [ 4361.183717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4361.185619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4361.188761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4361.190258] Lustre: Skipped 1 previous similar message [ 4365.848697] Lustre: server umount lustre-MDT0000 complete [ 4366.815848] LustreError: 213953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4366.821419] LustreError: 213953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4367.023266] LustreError: 213935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075469 with bad export cookie 5327544777912941856 [ 4367.026451] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4367.028812] LustreError: 213935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4367.140404] Lustre: server umount lustre-MDT0001 complete [ 4370.545843] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4371.515981] Key type lgssc unregistered [ 4371.633393] LNet: 217650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4371.636526] LNetError: 217650:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4371.646337] LNet: Removed LNI 192.168.201.133@tcp [ 4371.950548] Key type .llcrypt unregistered [ 4371.951794] Key type ._llcrypt unregistered [ 4378.878358] Key type ._llcrypt registered [ 4378.879643] Key type .llcrypt registered [ 4378.909908] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4379.211842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4379.264446] alg: No test for adler32 (adler32-zlib) [ 4380.111138] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 4380.197310] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4381.775154] Key type lgssc registered [ 4382.087662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4384.382500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4384.387528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4385.513840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4386.785024] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4389.319747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4389.426152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4390.804098] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4393.218637] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4395.014155] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4397.344318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4397.465881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.324336] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4401.541378] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4411.813484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4414.422735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4414.447487] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4414.518603] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4416.412046] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4418.672915] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4419.196501] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4429.761172] Lustre: Permanently deactivating lustre-OST0001 [ 4429.766561] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4429.908279] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4441.495556] Lustre: Permanently reactivating lustre-OST0001 [ 4441.501955] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4441.504715] Lustre: Skipped 2 previous similar messages [ 4451.309753] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4451.314678] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4454.587850] Lustre: server umount lustre-OST0001 complete [ 4455.904368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4455.906902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4455.910312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4456.419569] LustreError: 220712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4456.423228] LustreError: 220712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4461.537124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4461.537919] LustreError: 220712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4461.539655] Lustre: Skipped 3 previous similar messages [ 4466.656494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4466.657815] LustreError: 220709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4466.658631] Lustre: Skipped 1 previous similar message [ 4470.239109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4470.286968] Lustre: server umount lustre-OST0000 complete [ 4471.775146] LustreError: 222814:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 4471.776177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4471.782308] Lustre: Skipped 1 previous similar message [ 4471.784174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4476.383820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4476.386242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4481.014961] Lustre: server umount lustre-MDT0000 complete [ 4482.016303] LustreError: 218894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.020125] LustreError: 218894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4482.195361] LustreError: 220736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075584 with bad export cookie 14124725864277237566 [ 4482.198589] LustreError: 220736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4482.199084] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4482.318503] Lustre: server umount lustre-MDT0001 complete [ 4485.568141] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4486.563162] Key type lgssc unregistered [ 4486.682355] LNet: 223969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4486.684414] LNetError: 223969:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4486.692263] LNet: Removed LNI 192.168.201.133@tcp [ 4486.972478] Key type .llcrypt unregistered [ 4486.973441] Key type ._llcrypt unregistered [ 4500.208702] Key type ._llcrypt registered [ 4500.209600] Key type .llcrypt registered [ 4500.247937] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4500.617453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4500.664111] alg: No test for adler32 (adler32-zlib) [ 4501.517988] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 4501.605540] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4503.183149] Key type lgssc registered [ 4503.580801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4506.334499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4506.340282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4507.426671] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4507.434299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4507.492454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4508.881370] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4511.537131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4511.566273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4511.576131] Lustre: 226148:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4511.660060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4512.878554] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4515.248157] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4516.916373] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4519.094622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4519.169591] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4519.228220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4521.186911] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4523.422491] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4524.039230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 4530.210373] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4533.620807] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4541.191836] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4550.787526] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4561.592399] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4583.171693] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4583.174061] Lustre: Skipped 2 previous similar messages [ 4619.425480] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4619.428645] Lustre: Skipped 3 previous similar messages [ 4686.249179] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4686.251747] Lustre: Skipped 7 previous similar messages [ 4703.205875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4703.206357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4703.210200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4703.211515] LustreError: Skipped 1 previous similar message [ 4708.314100] Lustre: server umount lustre-OST0000 complete [ 4711.392259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4711.394983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4711.399628] Lustre: Skipped 1 previous similar message [ 4711.401824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4711.404543] Lustre: Skipped 1 previous similar message [ 4713.441027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4713.441287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4715.809422] Lustre: server umount lustre-MDT0000 complete [ 4717.126655] LustreError: 226138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075819 with bad export cookie 4024432272203003330 [ 4717.130686] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4717.130939] LustreError: 226138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4717.267825] Lustre: server umount lustre-MDT0001 complete [ 4721.025289] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4722.092548] Key type lgssc unregistered [ 4722.256524] LNet: 230961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4722.258981] LNetError: 230961:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4722.266497] LNet: Removed LNI 192.168.201.133@tcp [ 4722.642740] Key type .llcrypt unregistered [ 4722.643880] Key type ._llcrypt unregistered [ 4731.726620] Key type ._llcrypt registered [ 4731.727491] Key type .llcrypt registered [ 4731.760333] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4732.063597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4732.071752] alg: No test for adler32 (adler32-zlib) [ 4732.915268] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 4732.995117] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4734.583131] Key type lgssc registered [ 4734.960846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4737.571439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4737.577308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4738.712132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4740.005771] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4742.784090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4742.902462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4744.164757] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4746.396811] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4748.138217] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4750.348599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4750.461314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.400267] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4754.626885] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4763.810439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 4773.343737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4773.346306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4773.349766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4774.369062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4774.370558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4774.372077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.070381] Lustre: server umount lustre-OST0000 complete [ 4778.975754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4778.977379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4778.980818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4778.982206] Lustre: Skipped 1 previous similar message [ 4783.314100] Lustre: server umount lustre-MDT0000 complete [ 4784.460156] LustreError: 232331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075886 with bad export cookie 11952598448995669930 [ 4784.462543] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4784.464408] LustreError: 232331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4784.575967] Lustre: server umount lustre-MDT0001 complete [ 4787.934122] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4788.866979] Key type lgssc unregistered [ 4788.983378] LNet: 236055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4788.986039] LNetError: 236055:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4788.994274] LNet: Removed LNI 192.168.201.133@tcp [ 4789.296053] Key type .llcrypt unregistered [ 4789.297433] Key type ._llcrypt unregistered [ 4797.446516] Key type ._llcrypt registered [ 4797.447342] Key type .llcrypt registered [ 4797.480395] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4797.914471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4797.923440] alg: No test for adler32 (adler32-zlib) [ 4798.771552] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 4798.857033] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4800.439153] Key type lgssc registered [ 4800.765171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5064.674277] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 5065.633592] Key type lgssc unregistered [ 5065.757455] LNet: 238052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5065.759994] LNetError: 238052:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5065.769382] LNet: Removed LNI 192.168.201.133@tcp [ 5066.054364] Key type .llcrypt unregistered [ 5066.055728] Key type ._llcrypt unregistered [ 5073.480540] Key type ._llcrypt registered [ 5073.481734] Key type .llcrypt registered [ 5073.517647] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 5079.633975] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 5079.956298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5079.966523] alg: No test for adler32 (adler32-zlib) [ 5080.827041] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 5080.931745] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 5082.511123] Key type lgssc registered [ 5082.818486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5091.841375] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4770 sec ========== 06:56:33 (1783076193) [ 5092.363468] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:56:34 (1783076194) === [ 5093.470068] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:56:35 (1783076195) === [ 5105.652848] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 5106.575617] Key type lgssc unregistered [ 5106.689241] LNet: 243255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5106.690990] LNetError: 243255:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5106.696256] LNet: Removed LNI 192.168.201.133@tcp [ 5106.974214] Key type .llcrypt unregistered [ 5106.975254] Key type ._llcrypt unregistered